Oct 31 00:26:06.173666 ip-10-0-143-213 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 00:26:06.173677 ip-10-0-143-213 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 00:26:06.173685 ip-10-0-143-213 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 00:26:06.173910 ip-10-0-143-213 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 00:26:16.427756 ip-10-0-143-213 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 00:26:16.427773 ip-10-0-143-213 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 882d652527244f0ea603d7801e0111ec -- Oct 31 00:28:34.020048 ip-10-0-143-213 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:28:34.390279 ip-10-0-143-213 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:28:34.390279 ip-10-0-143-213 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:28:34.390279 ip-10-0-143-213 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:28:34.391691 ip-10-0-143-213 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:28:34.395808 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395713 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:34.395808 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395808 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395812 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395816 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395818 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395821 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395825 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395827 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395830 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395833 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395835 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395838 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395841 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395843 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395846 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395849 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395851 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395854 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395856 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395859 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395862 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:34.395875 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395865 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395871 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395874 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395877 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395880 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395883 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395886 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395890 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395892 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395895 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395898 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395900 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395903 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395906 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395908 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395911 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395914 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395916 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395919 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395921 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:34.396334 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395924 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395927 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395929 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395932 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395934 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395937 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395939 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395944 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395948 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395951 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395954 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395957 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395960 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395963 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395967 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395970 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395973 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395976 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395979 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:34.396822 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395982 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395985 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395988 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395990 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395993 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395996 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.395999 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396001 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396004 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396007 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396010 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396012 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396015 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396018 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396020 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396023 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396026 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396029 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396032 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396034 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:34.397261 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396037 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396039 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396042 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396044 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396047 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396050 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396054 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396057 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396059 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396062 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396065 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396069 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396073 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396076 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396079 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396082 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396084 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396087 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396090 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:34.397734 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396092 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396095 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396098 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396100 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396104 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396107 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396109 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396893 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396899 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396902 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396905 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396908 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396911 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396914 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396917 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396920 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396922 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396925 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396928 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396934 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:34.398186 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396937 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396940 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396943 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396946 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396949 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396951 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396954 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396957 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396960 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396962 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396965 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396968 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396971 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396973 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396976 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396979 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396981 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396984 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396986 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:34.398666 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396989 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396992 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396994 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396997 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.396999 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397002 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397005 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397007 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397010 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397012 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397015 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397018 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397023 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397026 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397028 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397031 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397034 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397036 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397039 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397041 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:34.399132 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397044 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397047 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397049 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397052 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397054 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397057 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397060 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397062 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397065 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397067 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397070 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397073 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397075 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397078 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397080 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397083 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397085 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397088 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397090 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:34.399614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397096 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397100 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397104 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397108 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397112 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397117 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397120 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397123 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397126 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397128 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397131 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397134 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397137 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397139 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397142 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397145 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397147 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397150 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397153 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397155 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:34.400084 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397158 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397160 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397163 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397165 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397168 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397170 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397173 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397176 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397178 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397181 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397183 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397186 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397189 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397192 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397195 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397676 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397682 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397686 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397690 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397693 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:34.400559 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397696 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397700 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397703 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397706 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397709 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397712 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397714 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397717 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397720 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397723 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397726 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397728 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397731 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397733 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397736 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397739 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397758 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397761 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397763 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397766 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:34.401043 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397769 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397772 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397774 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397777 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397779 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397782 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397785 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397788 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397791 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397795 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397797 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397800 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397803 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397806 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397809 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397811 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397816 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397820 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:34.401511 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397824 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397827 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397830 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397833 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397836 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397838 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397841 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397844 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397846 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397849 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397852 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397854 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397857 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397860 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397862 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397865 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397867 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397870 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397872 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:34.401984 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397875 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397878 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397880 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397883 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397887 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397890 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397892 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397895 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397898 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397901 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397903 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397906 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397908 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397911 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397914 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397917 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397919 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397922 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397924 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397927 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:34.402431 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397929 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397931 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397934 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397936 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397939 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397942 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397944 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397947 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397949 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397952 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397954 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397957 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397959 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397963 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397966 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397968 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397973 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397976 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397979 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397981 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:34.402909 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397984 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397987 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397989 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.397992 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398836 2575 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398853 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398861 2575 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398866 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398870 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398874 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398878 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398883 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398886 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398890 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398893 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398897 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398900 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398903 2575 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398908 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398911 2575 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398914 2575 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398917 2575 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398922 2575 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:28:34.403427 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398924 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398928 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398931 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398935 2575 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398939 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398943 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398946 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398950 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398953 2575 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398956 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398960 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398963 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398966 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398970 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398974 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398977 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398979 2575 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398983 2575 flags.go:64] FLAG: --enable-server="true" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398985 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398990 2575 flags.go:64] FLAG: --event-burst="100" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398994 2575 flags.go:64] FLAG: --event-qps="50" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.398997 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399000 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399003 2575 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:28:34.403998 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399007 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399010 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399013 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399018 2575 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399021 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399024 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399028 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399031 2575 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399034 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399037 2575 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399040 2575 flags.go:64] FLAG: --feature-gates="" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399045 2575 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399048 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399051 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399054 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399058 2575 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399061 2575 flags.go:64] FLAG: --help="false" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399064 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399068 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399072 2575 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399075 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399079 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399082 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:28:34.404602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399085 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399088 2575 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399091 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399094 2575 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399097 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399100 2575 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399103 2575 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399106 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399109 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399112 2575 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399115 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399118 2575 flags.go:64] FLAG: --lock-file="" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399123 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399126 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399129 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399135 2575 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399138 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399141 2575 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399145 2575 flags.go:64] FLAG: --logging-format="text" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399148 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399151 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399154 2575 flags.go:64] FLAG: --manifest-url="" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399157 2575 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399162 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399165 2575 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:28:34.405172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399169 2575 flags.go:64] FLAG: --max-pods="110" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399172 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399175 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399178 2575 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399182 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399185 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399188 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399191 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399199 2575 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399202 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399205 2575 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399209 2575 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399212 2575 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399214 2575 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399218 2575 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399221 2575 flags.go:64] FLAG: --port="10250" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399224 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399227 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-05ccdfc004f20af88" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399230 2575 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399235 2575 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399238 2575 flags.go:64] FLAG: --register-node="true" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399241 2575 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399245 2575 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399248 2575 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399250 2575 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399253 2575 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:28:34.405869 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399258 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399261 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399264 2575 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399267 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399270 2575 flags.go:64] FLAG: --runonce="false" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399274 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399277 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399280 2575 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399283 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399286 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399289 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399293 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399304 2575 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399307 2575 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399310 2575 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399314 2575 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399316 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399320 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399323 2575 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399325 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399331 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399334 2575 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399337 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399341 2575 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399344 2575 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:28:34.406953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399348 2575 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399352 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399355 2575 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399358 2575 flags.go:64] FLAG: --v="2" Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399362 2575 flags.go:64] FLAG: --version="false" Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399367 2575 flags.go:64] FLAG: --vmodule="" Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399371 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399375 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399511 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399516 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399520 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399523 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399525 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399528 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399531 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399533 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399536 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399539 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399541 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399545 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399548 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:34.407997 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399550 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399553 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399555 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399558 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399561 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399563 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399566 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399569 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399572 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399575 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399577 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399582 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399585 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399587 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399590 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399593 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399595 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399598 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399601 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:34.408868 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399603 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399608 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399610 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399613 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399616 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399619 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399622 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399625 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399628 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399630 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399633 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399635 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399639 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399642 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399645 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399648 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399650 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399653 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399656 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:34.409368 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399658 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399661 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399663 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399666 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399670 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399675 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399678 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399681 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399685 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399688 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399692 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399694 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399697 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399700 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399703 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399706 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399709 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399712 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399714 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:34.409980 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399717 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399720 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399722 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399725 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399728 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399730 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399733 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399736 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399739 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399756 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399759 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399761 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399764 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399767 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399770 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399773 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399776 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399779 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399783 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399785 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:34.411487 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399788 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:34.411969 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399791 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:34.411969 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399793 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:34.411969 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399796 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:34.411969 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399799 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:34.411969 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399802 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:34.411969 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399805 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:34.411969 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399809 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:34.411969 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399811 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:34.411969 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399814 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:34.411969 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399817 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:34.411969 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399819 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:34.411969 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399822 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:34.411969 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399825 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:34.411969 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399827 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:34.411969 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399830 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:34.411969 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.399857 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399900 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399904 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399907 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399911 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399913 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399916 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399919 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399922 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399925 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399927 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399930 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399933 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399935 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399939 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399942 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399945 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399948 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399950 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399953 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:34.412347 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399956 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399958 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399961 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399964 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399967 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399970 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399973 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399975 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399978 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399981 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399984 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399987 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399989 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399992 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399995 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.399997 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400000 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400003 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400006 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:34.412829 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400009 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400011 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400014 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400016 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400019 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400022 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400024 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400029 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400031 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400034 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400037 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400040 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400042 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400045 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400047 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400050 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400053 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400056 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400059 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:34.413280 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400062 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400064 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400069 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400073 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400076 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400080 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400084 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400088 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400092 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400095 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400098 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400101 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400105 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400107 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400110 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400112 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400115 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400118 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400120 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:34.413721 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400123 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400126 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400129 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400132 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400135 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400137 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400140 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400142 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400145 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400148 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400150 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400154 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400157 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400160 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400162 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400165 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400167 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400170 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400173 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:34.414178 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400175 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400179 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400181 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400184 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400186 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400189 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400192 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400195 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400199 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400201 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.400204 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.408618 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.408639 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.408655 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.408664 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409378 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409391 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409395 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409399 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409402 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:34.414614 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409405 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409408 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409411 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409414 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409417 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409420 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409423 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409427 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409429 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409433 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409436 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409438 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409441 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409444 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409447 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409450 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409452 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409455 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409458 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:34.415102 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409461 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409463 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409466 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409471 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409474 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409477 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409483 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409488 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409491 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409494 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409497 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409500 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409503 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409506 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409509 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409512 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409515 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409517 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409520 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:34.415551 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409523 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409526 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409528 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409531 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409534 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409536 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409539 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409542 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409544 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409550 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409553 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409558 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409560 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409563 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409566 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409570 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409575 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409577 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409580 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409583 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:34.416054 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409587 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409591 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409595 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409598 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409602 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409607 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409609 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409612 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409617 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409620 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409622 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409625 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409627 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409630 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409633 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409635 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409640 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409644 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409648 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:34.416525 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409650 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409653 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409656 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409659 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409662 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409664 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409667 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409670 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409672 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409675 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409678 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409680 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409683 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409686 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409689 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409691 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409694 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409697 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409699 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409702 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:34.417037 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409705 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:34.417492 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409707 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:34.417492 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409710 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:34.417492 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409712 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:34.417492 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.409757 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:28:34.417492 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409903 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:34.417492 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409907 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:34.417492 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409910 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:34.417492 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409912 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:34.417492 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409916 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:34.417492 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409919 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:34.417492 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409921 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:34.417492 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409924 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:34.417492 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409927 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:34.417492 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409930 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:34.417492 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409933 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:34.417492 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409936 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409939 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409941 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409944 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409946 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409949 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409952 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409954 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409957 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409960 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409962 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409965 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409967 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409970 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409973 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409975 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409978 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409980 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409983 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:34.417878 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409986 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409988 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409991 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409994 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.409997 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410000 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410002 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410005 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410008 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410010 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410013 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410015 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410018 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410021 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410024 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410026 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410029 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410033 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410037 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:34.418315 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410040 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410042 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410045 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410048 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410051 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410053 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410056 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410058 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410061 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410063 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410066 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410068 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410071 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410074 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410076 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410079 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410081 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410084 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410087 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:34.418763 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410089 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410092 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410094 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410097 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410100 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410103 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410105 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410108 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410111 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410113 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410116 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410118 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410121 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410124 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410126 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410129 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410133 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410136 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410139 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410142 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:34.419256 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410145 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:34.419728 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410148 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:34.419728 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410150 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:34.419728 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410153 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:34.419728 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410155 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:34.419728 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410158 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:34.419728 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410160 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:34.419728 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410163 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:34.419728 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410166 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:34.419728 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410168 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:34.419728 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410171 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:34.419728 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410173 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:34.419728 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410176 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:34.419728 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410178 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:34.419728 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410181 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:34.419728 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410184 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:34.419728 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410187 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:34.419728 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410189 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:34.420144 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.410213 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:28:34.420144 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410243 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:34.420144 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410247 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:34.420144 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410250 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:34.420144 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410253 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:34.420144 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410257 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:34.420144 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410260 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:34.420144 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410263 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:34.420144 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410266 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:34.420144 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410269 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:34.420144 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410272 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:34.420144 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410274 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:34.420144 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410277 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:34.420144 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410279 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:34.420144 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410282 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:34.420144 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410284 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410287 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410290 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410292 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410295 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410299 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410303 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410306 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410309 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410312 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410315 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410317 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410320 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410322 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410325 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410327 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410331 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410333 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410340 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:34.420516 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410342 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410345 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410348 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410350 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410353 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410356 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410358 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410361 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410364 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410367 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410370 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410372 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410375 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410378 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410380 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410383 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410386 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410389 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410392 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410395 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:34.421015 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410398 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410400 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410403 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410405 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410408 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410410 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410413 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410416 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410418 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410421 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410424 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410429 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410431 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410434 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410437 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410440 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410443 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410445 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410448 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:34.421468 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410450 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410453 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410456 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410458 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410461 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410464 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410466 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410469 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410471 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410474 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410476 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410479 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410482 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410484 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410487 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410490 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410495 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410498 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410501 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:34.421913 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410504 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410506 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410509 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410512 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410514 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410520 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410522 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410525 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410528 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410530 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410533 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410536 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410538 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410541 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:34.410543 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.410603 2575 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.415417 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.416444 2575 server.go:1014] "Starting client certificate rotation" Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.416539 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:28:34.422386 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.416580 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:28:34.422880 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.418769 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:28:34.422880 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.419932 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:28:34.432431 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.432410 2575 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:28:34.437874 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.437859 2575 log.go:25] "Validated CRI v1 image API" Oct 31 00:28:34.439594 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.439541 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:28:34.444081 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.444059 2575 fs.go:135] Filesystem UUIDs: map[567967fd-b706-463a-98d4-9a132fb073db:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fd096270-1e24-470b-bc7a-fa03b10dfec1:/dev/nvme0n1p4] Oct 31 00:28:34.444155 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.444080 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 00:28:34.450863 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.450724 2575 manager.go:217] Machine: {Timestamp:2026-10-31 00:28:34.448851264 +0000 UTC m=+0.323732945 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100324 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ea9daa5c10b1d293b1156d632f987 SystemUUID:ec2ea9da-a5c1-0b1d-293b-1156d632f987 BootID:882d6525-2724-4f0e-a603-d7801e0111ec Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d5:7b:03:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d5:7b:03:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:09:e1:a2:a0:bc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:28:34.450863 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.450855 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:28:34.451012 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.450966 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:28:34.454265 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.454240 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:28:34.454433 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.454266 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-213.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:28:34.454519 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.454438 2575 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:28:34.454519 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.454451 2575 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:28:34.454519 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.454497 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:28:34.454623 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.454524 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:28:34.454968 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.454950 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:28:34.455820 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.455802 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:28:34.460191 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.460177 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:28:34.460321 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.460310 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:28:34.460374 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.460350 2575 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:28:34.460374 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.460365 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:28:34.460467 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.460379 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:28:34.460467 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.460392 2575 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:28:34.460467 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.460405 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:28:34.460772 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.460758 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:28:34.463209 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.463192 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:28:34.470708 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.470693 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:28:34.471713 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.471699 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:28:34.471766 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.471731 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:28:34.471916 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.471905 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:28:34.471945 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.471922 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:28:34.471945 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.471928 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:28:34.471945 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.471934 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:28:34.471945 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.471941 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:28:34.472040 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.471952 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:28:34.472723 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.472715 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:28:34.472767 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.472724 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:28:34.472767 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.472731 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:28:34.472767 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.472737 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:28:34.472865 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.472779 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:28:34.472865 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.472788 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:28:34.473575 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.473566 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:28:34.473608 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.473578 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:28:34.477917 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.477906 2575 server.go:1263] "Started kubelet" Oct 31 00:28:34.478062 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.478034 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:28:34.478119 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.478030 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:28:34.478119 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.478096 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:28:34.478663 ip-10-0-143-213 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:28:34.479758 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.479488 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:28:34.479758 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.479567 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-213.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 00:28:34.479990 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.479973 2575 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:28:34.484803 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.484780 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 00:28:34.486965 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.486825 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:28:34.486965 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.486839 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:28:34.487100 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.487055 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-213.us-east-2.compute.internal\" not found" Oct 31 00:28:34.487611 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.487592 2575 factory.go:55] Registering systemd factory Oct 31 00:28:34.487611 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.487612 2575 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:28:34.487718 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.487635 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-213.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 00:28:34.487778 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.487736 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 00:28:34.487888 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.487873 2575 factory.go:153] Registering CRI-O factory Oct 31 00:28:34.487888 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.487887 2575 factory.go:223] Registration of the crio container factory successfully Oct 31 00:28:34.487980 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.487890 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:28:34.487980 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.487939 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:28:34.487980 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.487959 2575 factory.go:103] Registering Raw factory Oct 31 00:28:34.487980 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.487973 2575 manager.go:1199] Started watching for new ooms in manager Oct 31 00:28:34.488165 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.488060 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:28:34.488165 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.488071 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:28:34.488253 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.488175 2575 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:28:34.488253 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.488183 2575 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:28:34.491192 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.487777 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-213.us-east-2.compute.internal.18e375b91953319e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-213.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-213.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:34.477871518 +0000 UTC m=+0.352753193,LastTimestamp:2026-10-31 00:28:34.477871518 +0000 UTC m=+0.352753193,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-213.us-east-2.compute.internal,}" Oct 31 00:28:34.491452 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.491434 2575 manager.go:319] Starting recovery of all containers Oct 31 00:28:34.495336 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.495301 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-213.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 00:28:34.495336 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.495313 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-85t6q" Oct 31 00:28:34.500855 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.500829 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 00:28:34.501658 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.501638 2575 manager.go:324] Recovery completed Oct 31 00:28:34.507652 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.507519 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-85t6q" Oct 31 00:28:34.507652 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.507628 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:28:34.509804 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.509786 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:28:34.509883 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.509821 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:28:34.509883 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.509836 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:28:34.510385 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.510362 2575 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:28:34.510385 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.510372 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:28:34.510484 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.510374 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:28:34.510484 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.510439 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:28:34.511160 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.511143 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:28:34.511228 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.511169 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:28:34.511228 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.511180 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:28:34.511228 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.511157 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-213.us-east-2.compute.internal.18e375b91b3a71e5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-213.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-213.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-213.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:34.509804005 +0000 UTC m=+0.384685686,LastTimestamp:2026-10-31 00:28:34.509804005 +0000 UTC m=+0.384685686,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-213.us-east-2.compute.internal,}" Oct 31 00:28:34.512536 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.512518 2575 policy_none.go:50] "Start" Oct 31 00:28:34.512536 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.512536 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:28:34.512641 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.512546 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:28:34.514576 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.514565 2575 policy_none.go:44] "Start" Oct 31 00:28:34.551097 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.551057 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:28:34.553886 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.553868 2575 manager.go:341] "Starting Device Plugin manager" Oct 31 00:28:34.553982 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.553904 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:28:34.553982 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.553914 2575 server.go:87] "Starting device plugin registration server" Oct 31 00:28:34.554202 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.554184 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:28:34.554268 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.554203 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:28:34.554316 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.554302 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:28:34.554644 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.554388 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:28:34.554644 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.554398 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:28:34.555037 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.555017 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:28:34.555100 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.555070 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-213.us-east-2.compute.internal\" not found" Oct 31 00:28:34.655099 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.654875 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:28:34.656111 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.656092 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:28:34.656215 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.656121 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:28:34.656215 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.656131 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:28:34.656215 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.656158 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:34.666622 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.666602 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:34.666716 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.666627 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-213.us-east-2.compute.internal\": node \"ip-10-0-143-213.us-east-2.compute.internal\" not found" Oct 31 00:28:34.685530 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.685509 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-213.us-east-2.compute.internal\" not found" Oct 31 00:28:34.687429 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.687402 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:28:34.688546 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.688531 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:28:34.688601 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.688553 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:28:34.688601 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.688573 2575 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:28:34.688688 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.688603 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 00:28:34.695416 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.695397 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:28:34.786614 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.786575 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-213.us-east-2.compute.internal\" not found" Oct 31 00:28:34.788868 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.788831 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal"] Oct 31 00:28:34.788939 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.788908 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:28:34.789901 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.789887 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:28:34.789947 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.789915 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:28:34.790006 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.789948 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:28:34.792189 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.792176 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:28:34.792351 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.792337 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:34.792393 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.792388 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:28:34.792960 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.792944 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:28:34.793027 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.792973 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:28:34.793027 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.792983 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:28:34.793027 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.792944 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:28:34.793124 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.793048 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:28:34.793124 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.793066 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:28:34.795201 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.795184 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:34.795309 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.795208 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:28:34.795971 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.795957 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:28:34.796043 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.795976 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:28:34.796043 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.795985 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:28:34.814273 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.814253 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-213.us-east-2.compute.internal\" not found" node="ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:34.816486 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.816471 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-213.us-east-2.compute.internal\" not found" node="ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:34.886867 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.886823 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-213.us-east-2.compute.internal\" not found" Oct 31 00:28:34.889070 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.889049 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb8ff3cdd197b5db95c89f56a4f8fb1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"bb8ff3cdd197b5db95c89f56a4f8fb1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:34.889133 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.889080 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b347d22e86cd58c1b6312f0f07ef259-config\") pod \"kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"2b347d22e86cd58c1b6312f0f07ef259\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:34.889133 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.889097 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb8ff3cdd197b5db95c89f56a4f8fb1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"bb8ff3cdd197b5db95c89f56a4f8fb1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:34.987549 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:34.987482 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-213.us-east-2.compute.internal\" not found" Oct 31 00:28:34.989685 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.989671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb8ff3cdd197b5db95c89f56a4f8fb1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"bb8ff3cdd197b5db95c89f56a4f8fb1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:34.989740 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.989695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb8ff3cdd197b5db95c89f56a4f8fb1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"bb8ff3cdd197b5db95c89f56a4f8fb1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:34.989740 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.989715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b347d22e86cd58c1b6312f0f07ef259-config\") pod \"kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"2b347d22e86cd58c1b6312f0f07ef259\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:34.989827 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.989794 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b347d22e86cd58c1b6312f0f07ef259-config\") pod \"kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"2b347d22e86cd58c1b6312f0f07ef259\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:34.989827 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.989811 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb8ff3cdd197b5db95c89f56a4f8fb1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"bb8ff3cdd197b5db95c89f56a4f8fb1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:34.989899 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:34.989794 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb8ff3cdd197b5db95c89f56a4f8fb1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"bb8ff3cdd197b5db95c89f56a4f8fb1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:35.087600 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:35.087560 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-213.us-east-2.compute.internal\" not found" Oct 31 00:28:35.116986 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:35.116962 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:35.119092 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:35.119075 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:35.188069 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:35.188036 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-213.us-east-2.compute.internal\" not found" Oct 31 00:28:36.601473 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:36.601403 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-213.us-east-2.compute.internal\" not found" Oct 31 00:28:36.701889 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:36.701863 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-213.us-east-2.compute.internal\" not found" Oct 31 00:28:36.729156 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:36.729133 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 00:28:36.729579 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:36.729271 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:28:36.802714 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:36.802689 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-213.us-east-2.compute.internal\" not found" Oct 31 00:28:36.827141 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:36.827090 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:23:35 +0000 UTC" deadline="2028-07-16 07:04:19.621020798 +0000 UTC" Oct 31 00:28:36.827141 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:36.827133 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14982h35m42.793890004s" Oct 31 00:28:36.903680 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:36.903638 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-213.us-east-2.compute.internal\" not found" Oct 31 00:28:37.003735 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:37.003705 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-213.us-east-2.compute.internal\" not found" Oct 31 00:28:37.056167 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:37.056139 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8ff3cdd197b5db95c89f56a4f8fb1a.slice/crio-34cac5d6a5a3bffaec384bbcac6431fd3ab274264163a27a4188dd9723401e3a WatchSource:0}: Error finding container 34cac5d6a5a3bffaec384bbcac6431fd3ab274264163a27a4188dd9723401e3a: Status 404 returned error can't find the container with id 34cac5d6a5a3bffaec384bbcac6431fd3ab274264163a27a4188dd9723401e3a Oct 31 00:28:37.057240 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:37.057220 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b347d22e86cd58c1b6312f0f07ef259.slice/crio-e768af5000c0dcee47d0b3b7dafcfd78487ca2a82c5a03555e371bb361caa13d WatchSource:0}: Error finding container e768af5000c0dcee47d0b3b7dafcfd78487ca2a82c5a03555e371bb361caa13d: Status 404 returned error can't find the container with id e768af5000c0dcee47d0b3b7dafcfd78487ca2a82c5a03555e371bb361caa13d Oct 31 00:28:37.059753 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.059735 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:28:37.062215 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.062201 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:28:37.101081 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.101062 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:37.114932 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.114913 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:28:37.116602 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.116586 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:28:37.126251 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.126232 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:28:37.207011 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.206948 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:28:37.232145 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.232125 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:28:37.758474 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.758443 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:28:37.774697 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.774660 2575 apiserver.go:52] "Watching apiserver" Oct 31 00:28:37.786963 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.786209 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:28:37.786963 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.786576 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-dcljc","openshift-multus/network-metrics-daemon-jcdhx","openshift-network-operator/iptables-alerter-sm6rp","kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal","openshift-multus/multus-pghrd","openshift-ovn-kubernetes/ovnkube-node-dhwx4","kube-system/konnectivity-agent-b64jh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst","openshift-cluster-node-tuning-operator/tuned-dwxbv","openshift-dns/node-resolver-wvff6","openshift-network-diagnostics/network-check-target-zwhbr","kube-system/global-pull-secret-syncer-dvl2d","openshift-image-registry/node-ca-9l52l"] Oct 31 00:28:37.789117 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.789090 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:37.789228 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:37.789175 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:28:37.791425 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.791403 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.793975 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.793954 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:28:37.796619 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.795919 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.798972 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.798953 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.799755 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.799733 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:28:37.799857 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.799802 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:28:37.799906 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.799891 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:28:37.799956 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.799928 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:28:37.799991 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.799982 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:28:37.800649 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.800409 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cnz4r\"" Oct 31 00:28:37.800649 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.800494 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 00:28:37.803361 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.802528 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dnwdb\"" Oct 31 00:28:37.803361 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.802682 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:28:37.803361 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.802797 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:28:37.803361 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.802834 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:28:37.803361 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.803043 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:28:37.803361 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.803152 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:28:37.803361 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.803209 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:28:37.803361 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.803227 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-56d49\"" Oct 31 00:28:37.803361 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.803325 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:28:37.803811 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.803377 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:28:37.804659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.803951 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b64jh" Oct 31 00:28:37.807702 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.807680 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:37.811035 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811011 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-slash\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.811129 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-node-log\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.811129 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811073 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-system-cni-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.811129 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811099 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-cni-bin\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.811296 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-cni-multus\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.811296 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811174 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c892baf2-ce51-4c47-8d79-f75f4f50c09d-konnectivity-ca\") pod \"konnectivity-agent-b64jh\" (UID: \"c892baf2-ce51-4c47-8d79-f75f4f50c09d\") " pod="kube-system/konnectivity-agent-b64jh" Oct 31 00:28:37.811296 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811212 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8mpp\" (UniqueName: \"kubernetes.io/projected/ba178af9-0f61-4c7b-a5ab-36e772495016-kube-api-access-b8mpp\") pod \"network-metrics-daemon-jcdhx\" (UID: \"ba178af9-0f61-4c7b-a5ab-36e772495016\") " pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:37.811296 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-run-netns\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.811296 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811271 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-etc-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.811296 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811293 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.811692 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811316 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-cni-netd\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.811692 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811339 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-cni-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.811692 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811364 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3073e91a-8363-4d2a-8774-635302d4b040-cni-binary-copy\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.811692 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811388 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3073e91a-8363-4d2a-8774-635302d4b040-multus-daemon-config\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.811692 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811427 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs\") pod \"network-metrics-daemon-jcdhx\" (UID: \"ba178af9-0f61-4c7b-a5ab-36e772495016\") " pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:37.811692 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811449 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-systemd\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.811692 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811472 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-var-lib-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.811692 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811557 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-system-cni-dir\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.811692 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811588 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-cni-binary-copy\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.811692 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811622 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.812067 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-kubelet\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.812067 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811748 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-ovn\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.812067 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811781 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh2g8\" (UniqueName: \"kubernetes.io/projected/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-kube-api-access-xh2g8\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.812067 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811806 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8dl6\" (UniqueName: \"kubernetes.io/projected/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-kube-api-access-q8dl6\") pod \"iptables-alerter-sm6rp\" (UID: \"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0\") " pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:28:37.812067 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811931 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-k8s-cni-cncf-io\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.812067 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811962 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-log-socket\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.812067 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.811988 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-netns\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.812067 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812016 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-kubelet\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.812067 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812042 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.812422 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812099 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-env-overrides\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.812422 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812129 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-host-slash\") pod \"iptables-alerter-sm6rp\" (UID: \"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0\") " pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:28:37.812422 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812162 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-multus-certs\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.812422 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812211 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-etc-kubernetes\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.812422 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wvxm\" (UniqueName: \"kubernetes.io/projected/3073e91a-8363-4d2a-8774-635302d4b040-kube-api-access-4wvxm\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.812422 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812316 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.812422 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812349 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26png\" (UniqueName: \"kubernetes.io/projected/6dca6aeb-205b-4634-9216-a98ba3957025-kube-api-access-26png\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.812422 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812389 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovnkube-config\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.812422 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812408 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovn-node-metrics-cert\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.812422 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-iptables-alerter-script\") pod \"iptables-alerter-sm6rp\" (UID: \"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0\") " pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:28:37.812761 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812448 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-cnibin\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.812761 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812483 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-socket-dir-parent\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.812761 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812507 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-hostroot\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.812761 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812547 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.812761 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812573 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-cni-bin\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.812761 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.812761 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812638 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovnkube-script-lib\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.812761 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812680 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-conf-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.812761 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812728 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-cnibin\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.812761 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812760 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c892baf2-ce51-4c47-8d79-f75f4f50c09d-agent-certs\") pod \"konnectivity-agent-b64jh\" (UID: \"c892baf2-ce51-4c47-8d79-f75f4f50c09d\") " pod="kube-system/konnectivity-agent-b64jh" Oct 31 00:28:37.813214 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.812787 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-systemd-units\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.813910 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.813883 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:37.816552 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.816535 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wvff6" Oct 31 00:28:37.820588 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.820571 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:37.820694 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:37.820640 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:28:37.821598 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.821578 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:28:37.822900 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.822883 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:37.822995 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:37.822962 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:28:37.825236 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.825219 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:28:37.849309 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.848264 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rxs4m" Oct 31 00:28:37.859914 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.859889 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rxs4m" Oct 31 00:28:37.900057 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.900024 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:28:37.900203 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.900024 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:28:37.900203 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.900107 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:28:37.900338 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.900323 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:28:37.900371 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.900328 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:28:37.900433 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.900414 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:28:37.900479 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.900456 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:28:37.900514 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.900497 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:28:37.900564 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.900549 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:28:37.900939 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.900626 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:28:37.900939 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.900701 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:28:37.901044 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.900971 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wzwtm\"" Oct 31 00:28:37.902106 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.902085 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:28:37.902455 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.902429 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:28:37.905196 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.902638 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:28:37.905196 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.902711 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j8g45\"" Oct 31 00:28:37.905196 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.902768 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9bm5h\"" Oct 31 00:28:37.905196 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.903019 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:28:37.905196 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.903029 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b2pkn\"" Oct 31 00:28:37.905196 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.902644 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-g4hxn\"" Oct 31 00:28:37.905196 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.903300 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r62gb\"" Oct 31 00:28:37.913016 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.912988 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysctl-conf\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:37.913120 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913027 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.913120 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913054 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3073e91a-8363-4d2a-8774-635302d4b040-multus-daemon-config\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.913224 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913115 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-device-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:37.913224 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913157 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcmn7\" (UniqueName: \"kubernetes.io/projected/1db4e02b-1e7a-4361-994e-f30507dbf0e7-kube-api-access-gcmn7\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:37.913224 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-systemd\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.913224 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-var-lib-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.913417 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-cni-binary-copy\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.913417 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913267 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2lm\" (UniqueName: \"kubernetes.io/projected/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-kube-api-access-lr2lm\") pod \"node-resolver-wvff6\" (UID: \"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa\") " pod="openshift-dns/node-resolver-wvff6" Oct 31 00:28:37.913417 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913290 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79b89acd-a759-4036-bd91-16ee9ad9e288-host\") pod \"node-ca-9l52l\" (UID: \"79b89acd-a759-4036-bd91-16ee9ad9e288\") " pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:28:37.913417 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-kubelet\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.913417 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913340 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8dl6\" (UniqueName: \"kubernetes.io/projected/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-kube-api-access-q8dl6\") pod \"iptables-alerter-sm6rp\" (UID: \"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0\") " pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:28:37.913417 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913366 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-k8s-cni-cncf-io\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.913417 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913391 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-tuned\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:37.913417 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913413 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-hosts-file\") pod \"node-resolver-wvff6\" (UID: \"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa\") " pod="openshift-dns/node-resolver-wvff6" Oct 31 00:28:37.913814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913436 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7lkc\" (UniqueName: \"kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc\") pod \"network-check-target-zwhbr\" (UID: \"6f86ab44-a957-4e07-8bdf-5a8f813f27d4\") " pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:37.913814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-log-socket\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.913814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-netns\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.913814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.913814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913536 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-multus-certs\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.913814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913560 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-etc-kubernetes\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.913814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913582 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.913814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26png\" (UniqueName: \"kubernetes.io/projected/6dca6aeb-205b-4634-9216-a98ba3957025-kube-api-access-26png\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.913814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913629 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-lib-modules\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:37.913814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913634 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3073e91a-8363-4d2a-8774-635302d4b040-multus-daemon-config\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.913814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913652 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8cjz\" (UniqueName: \"kubernetes.io/projected/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-kube-api-access-j8cjz\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:37.913814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913704 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.913814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913726 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:37.913814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913760 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-netns\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.913814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913773 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-systemd\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.913814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.913796 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-var-lib-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.914320 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914169 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.914320 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914207 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-multus-certs\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.914320 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914231 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-etc-kubernetes\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.914320 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914249 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-cni-binary-copy\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.914320 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914290 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-kubelet\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.914489 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914321 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.914523 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914483 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-k8s-cni-cncf-io\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.914556 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914538 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovnkube-config\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.914594 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914577 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-iptables-alerter-script\") pod \"iptables-alerter-sm6rp\" (UID: \"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0\") " pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:28:37.914646 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-socket-dir-parent\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.914705 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914659 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-hostroot\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.914705 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-registration-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:37.914784 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914747 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:37.914784 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914771 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-tmp-dir\") pod \"node-resolver-wvff6\" (UID: \"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa\") " pod="openshift-dns/node-resolver-wvff6" Oct 31 00:28:37.914860 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.914860 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914847 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-cni-bin\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.914955 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovnkube-script-lib\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.914955 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914896 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-conf-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.914955 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914944 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-cnibin\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.915147 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914968 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c892baf2-ce51-4c47-8d79-f75f4f50c09d-konnectivity-ca\") pod \"konnectivity-agent-b64jh\" (UID: \"c892baf2-ce51-4c47-8d79-f75f4f50c09d\") " pod="kube-system/konnectivity-agent-b64jh" Oct 31 00:28:37.915147 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.914994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-systemd-units\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.915147 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915019 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-slash\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.915147 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915031 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovnkube-config\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.915147 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915045 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-cni-bin\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.915147 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915072 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-sys-fs\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:37.915147 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915089 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-conf-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.915147 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915097 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-kubernetes\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:37.915147 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8mpp\" (UniqueName: \"kubernetes.io/projected/ba178af9-0f61-4c7b-a5ab-36e772495016-kube-api-access-b8mpp\") pod \"network-metrics-daemon-jcdhx\" (UID: \"ba178af9-0f61-4c7b-a5ab-36e772495016\") " pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:37.915561 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915217 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-run-netns\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.915561 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-etc-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.915561 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915273 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-cni-netd\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.915561 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915297 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-cni-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.915561 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915320 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3073e91a-8363-4d2a-8774-635302d4b040-cni-binary-copy\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.915561 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915347 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-tmp\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:37.915561 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs\") pod \"network-metrics-daemon-jcdhx\" (UID: \"ba178af9-0f61-4c7b-a5ab-36e772495016\") " pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:37.915561 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915399 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-system-cni-dir\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.915561 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915426 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.915561 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915451 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79b89acd-a759-4036-bd91-16ee9ad9e288-serviceca\") pod \"node-ca-9l52l\" (UID: \"79b89acd-a759-4036-bd91-16ee9ad9e288\") " pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:28:37.915561 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915477 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdn2m\" (UniqueName: \"kubernetes.io/projected/79b89acd-a759-4036-bd91-16ee9ad9e288-kube-api-access-mdn2m\") pod \"node-ca-9l52l\" (UID: \"79b89acd-a759-4036-bd91-16ee9ad9e288\") " pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:28:37.915561 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915501 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-ovn\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.915561 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915526 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh2g8\" (UniqueName: \"kubernetes.io/projected/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-kube-api-access-xh2g8\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.915561 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915543 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-iptables-alerter-script\") pod \"iptables-alerter-sm6rp\" (UID: \"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0\") " pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:28:37.915561 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915551 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-var-lib-kubelet\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:37.916246 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915577 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7b81351-0c71-49b6-bd86-e48483b8b555-kubelet-config\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:37.916246 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915594 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-hostroot\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.916246 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915635 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.916246 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915684 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-cni-netd\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.916246 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.915721 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-cni-bin\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.916246 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.916036 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-cni-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.916246 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.916131 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovnkube-script-lib\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.916246 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.916182 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-cnibin\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.916607 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.916466 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3073e91a-8363-4d2a-8774-635302d4b040-cni-binary-copy\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.916607 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:37.916552 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:28:37.916720 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:37.916611 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs podName:ba178af9-0f61-4c7b-a5ab-36e772495016 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:38.416597433 +0000 UTC m=+2.978556869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs") pod "network-metrics-daemon-jcdhx" (UID: "ba178af9-0f61-4c7b-a5ab-36e772495016") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:28:37.916720 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.916655 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c892baf2-ce51-4c47-8d79-f75f4f50c09d-konnectivity-ca\") pod \"konnectivity-agent-b64jh\" (UID: \"c892baf2-ce51-4c47-8d79-f75f4f50c09d\") " pod="kube-system/konnectivity-agent-b64jh" Oct 31 00:28:37.916824 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.916722 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-systemd-units\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.916824 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.916761 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-slash\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.916824 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.916798 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-cni-bin\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.916824 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.916805 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-system-cni-dir\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.917005 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.916844 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-log-socket\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.917005 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.916849 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-socket-dir-parent\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.917005 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.916880 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-kubelet\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.917005 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.916910 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-socket-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:37.917005 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.916937 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-env-overrides\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.917005 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.916963 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-host-slash\") pod \"iptables-alerter-sm6rp\" (UID: \"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0\") " pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:28:37.917273 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.917033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wvxm\" (UniqueName: \"kubernetes.io/projected/3073e91a-8363-4d2a-8774-635302d4b040-kube-api-access-4wvxm\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.917273 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.917061 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:37.917273 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.917087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-modprobe-d\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:37.917273 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.917111 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-systemd\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:37.917273 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.917139 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovn-node-metrics-cert\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.917273 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.917171 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-cnibin\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.917273 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.917196 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysconfig\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:37.917273 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.917220 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7b81351-0c71-49b6-bd86-e48483b8b555-dbus\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:37.917273 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.917250 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.917273 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.917261 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-host-slash\") pod \"iptables-alerter-sm6rp\" (UID: \"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0\") " pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:28:37.917728 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.917308 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-ovn\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.917728 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.917322 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-run-netns\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.917728 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.917355 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-etc-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.917728 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.917386 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-kubelet\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.917728 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.917631 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:28:37.917728 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.917723 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-env-overrides\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.918108 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.918084 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-cnibin\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.918246 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.918148 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.918246 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.918184 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c892baf2-ce51-4c47-8d79-f75f4f50c09d-agent-certs\") pod \"konnectivity-agent-b64jh\" (UID: \"c892baf2-ce51-4c47-8d79-f75f4f50c09d\") " pod="kube-system/konnectivity-agent-b64jh" Oct 31 00:28:37.918246 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.918212 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-run\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:37.918246 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.918236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-sys\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:37.918442 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.918251 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-host\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:37.918442 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.918267 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-node-log\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.918442 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.918312 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-system-cni-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.918442 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.918340 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-cni-multus\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.918442 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.918356 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysctl-d\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:37.918442 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.918403 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-node-log\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.918442 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.918427 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.919193 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.919169 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-cni-multus\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.919273 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.919231 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-system-cni-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.922234 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.922211 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c892baf2-ce51-4c47-8d79-f75f4f50c09d-agent-certs\") pod \"konnectivity-agent-b64jh\" (UID: \"c892baf2-ce51-4c47-8d79-f75f4f50c09d\") " pod="kube-system/konnectivity-agent-b64jh" Oct 31 00:28:37.922468 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.922447 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovn-node-metrics-cert\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:37.929633 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.929604 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26png\" (UniqueName: \"kubernetes.io/projected/6dca6aeb-205b-4634-9216-a98ba3957025-kube-api-access-26png\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:37.931240 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.931159 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8dl6\" (UniqueName: \"kubernetes.io/projected/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-kube-api-access-q8dl6\") pod \"iptables-alerter-sm6rp\" (UID: \"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0\") " pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:28:37.931240 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.931207 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8mpp\" (UniqueName: \"kubernetes.io/projected/ba178af9-0f61-4c7b-a5ab-36e772495016-kube-api-access-b8mpp\") pod \"network-metrics-daemon-jcdhx\" (UID: \"ba178af9-0f61-4c7b-a5ab-36e772495016\") " pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:37.933161 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.933142 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wvxm\" (UniqueName: \"kubernetes.io/projected/3073e91a-8363-4d2a-8774-635302d4b040-kube-api-access-4wvxm\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:28:37.936681 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:37.936642 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh2g8\" (UniqueName: \"kubernetes.io/projected/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-kube-api-access-xh2g8\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:38.006989 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.006934 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" event={"ID":"bb8ff3cdd197b5db95c89f56a4f8fb1a","Type":"ContainerStarted","Data":"34cac5d6a5a3bffaec384bbcac6431fd3ab274264163a27a4188dd9723401e3a"} Oct 31 00:28:38.008327 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.008277 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" event={"ID":"2b347d22e86cd58c1b6312f0f07ef259","Type":"ContainerStarted","Data":"e768af5000c0dcee47d0b3b7dafcfd78487ca2a82c5a03555e371bb361caa13d"} Oct 31 00:28:38.018891 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.018825 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-tmp\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.018891 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.018867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79b89acd-a759-4036-bd91-16ee9ad9e288-serviceca\") pod \"node-ca-9l52l\" (UID: \"79b89acd-a759-4036-bd91-16ee9ad9e288\") " pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:28:38.018891 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.018885 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdn2m\" (UniqueName: \"kubernetes.io/projected/79b89acd-a759-4036-bd91-16ee9ad9e288-kube-api-access-mdn2m\") pod \"node-ca-9l52l\" (UID: \"79b89acd-a759-4036-bd91-16ee9ad9e288\") " pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:28:38.019088 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.018909 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-var-lib-kubelet\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019088 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.018931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7b81351-0c71-49b6-bd86-e48483b8b555-kubelet-config\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:38.019088 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.018953 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-socket-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:38.019088 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.018978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:38.019088 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.018999 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-modprobe-d\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019088 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-systemd\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019088 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysconfig\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019088 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7b81351-0c71-49b6-bd86-e48483b8b555-dbus\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:38.019088 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019086 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-run\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019430 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019107 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-sys\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019430 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-host\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019430 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysctl-d\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019430 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019178 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysctl-conf\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019430 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019203 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-device-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:38.019430 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019228 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcmn7\" (UniqueName: \"kubernetes.io/projected/1db4e02b-1e7a-4361-994e-f30507dbf0e7-kube-api-access-gcmn7\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:38.019430 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019255 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2lm\" (UniqueName: \"kubernetes.io/projected/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-kube-api-access-lr2lm\") pod \"node-resolver-wvff6\" (UID: \"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa\") " pod="openshift-dns/node-resolver-wvff6" Oct 31 00:28:38.019430 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019330 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysconfig\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019430 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019364 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79b89acd-a759-4036-bd91-16ee9ad9e288-serviceca\") pod \"node-ca-9l52l\" (UID: \"79b89acd-a759-4036-bd91-16ee9ad9e288\") " pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:28:38.019430 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019393 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:38.019430 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019429 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-var-lib-kubelet\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019430 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019427 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7b81351-0c71-49b6-bd86-e48483b8b555-kubelet-config\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:38.019993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019487 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-systemd\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019528 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-socket-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:38.019993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019564 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7b81351-0c71-49b6-bd86-e48483b8b555-dbus\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:38.019993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79b89acd-a759-4036-bd91-16ee9ad9e288-host\") pod \"node-ca-9l52l\" (UID: \"79b89acd-a759-4036-bd91-16ee9ad9e288\") " pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:28:38.019993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019601 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-modprobe-d\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019611 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-tuned\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019602 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79b89acd-a759-4036-bd91-16ee9ad9e288-host\") pod \"node-ca-9l52l\" (UID: \"79b89acd-a759-4036-bd91-16ee9ad9e288\") " pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:28:38.019993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-hosts-file\") pod \"node-resolver-wvff6\" (UID: \"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa\") " pod="openshift-dns/node-resolver-wvff6" Oct 31 00:28:38.019993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019654 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lkc\" (UniqueName: \"kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc\") pod \"network-check-target-zwhbr\" (UID: \"6f86ab44-a957-4e07-8bdf-5a8f813f27d4\") " pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:38.019993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019688 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-device-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:38.019993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019714 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysctl-d\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-lib-modules\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8cjz\" (UniqueName: \"kubernetes.io/projected/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-kube-api-access-j8cjz\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019775 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:38.019993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019803 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-registration-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:38.019993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019813 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-lib-modules\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.019993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019823 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysctl-conf\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.020568 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019827 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:38.020568 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019868 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-tmp-dir\") pod \"node-resolver-wvff6\" (UID: \"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa\") " pod="openshift-dns/node-resolver-wvff6" Oct 31 00:28:38.020568 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019895 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:38.020568 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019898 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-host\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.020568 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019904 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-sys-fs\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:38.020568 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019957 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-sys-fs\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:38.020568 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.019941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-kubernetes\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.020568 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:38.019978 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:28:38.020568 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:38.020078 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret podName:e7b81351-0c71-49b6-bd86-e48483b8b555 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:38.520059909 +0000 UTC m=+3.082019331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret") pod "global-pull-secret-syncer-dvl2d" (UID: "e7b81351-0c71-49b6-bd86-e48483b8b555") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:28:38.020568 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.020199 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-run\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.020568 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.020235 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-registration-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:38.020568 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.020259 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-sys\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.020568 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.020294 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-hosts-file\") pod \"node-resolver-wvff6\" (UID: \"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa\") " pod="openshift-dns/node-resolver-wvff6" Oct 31 00:28:38.020568 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.020306 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-kubernetes\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.021221 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.020763 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-tmp-dir\") pod \"node-resolver-wvff6\" (UID: \"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa\") " pod="openshift-dns/node-resolver-wvff6" Oct 31 00:28:38.022206 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.022181 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-tmp\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.022400 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.022349 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-tuned\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.032470 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:38.032449 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:28:38.032578 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:38.032477 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:28:38.032578 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:38.032490 2575 projected.go:196] Error preparing data for projected volume kube-api-access-f7lkc for pod openshift-network-diagnostics/network-check-target-zwhbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:28:38.032578 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:38.032547 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc podName:6f86ab44-a957-4e07-8bdf-5a8f813f27d4 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:38.532529195 +0000 UTC m=+3.094488619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f7lkc" (UniqueName: "kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc") pod "network-check-target-zwhbr" (UID: "6f86ab44-a957-4e07-8bdf-5a8f813f27d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:28:38.035368 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.035338 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdn2m\" (UniqueName: \"kubernetes.io/projected/79b89acd-a759-4036-bd91-16ee9ad9e288-kube-api-access-mdn2m\") pod \"node-ca-9l52l\" (UID: \"79b89acd-a759-4036-bd91-16ee9ad9e288\") " pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:28:38.035525 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.035500 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2lm\" (UniqueName: \"kubernetes.io/projected/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-kube-api-access-lr2lm\") pod \"node-resolver-wvff6\" (UID: \"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa\") " pod="openshift-dns/node-resolver-wvff6" Oct 31 00:28:38.036590 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.036562 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcmn7\" (UniqueName: \"kubernetes.io/projected/1db4e02b-1e7a-4361-994e-f30507dbf0e7-kube-api-access-gcmn7\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:38.039333 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.039311 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8cjz\" (UniqueName: \"kubernetes.io/projected/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-kube-api-access-j8cjz\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.123928 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.123903 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:28:38.128816 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.128789 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:38.136035 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:38.136003 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2f6cc8_7a15_44cd_99c1_53ee713c3fb0.slice/crio-2dfc3f7ccf8877dde406b15a8d42f32ff79e2c4fe1235e03218ef60570730b5e WatchSource:0}: Error finding container 2dfc3f7ccf8877dde406b15a8d42f32ff79e2c4fe1235e03218ef60570730b5e: Status 404 returned error can't find the container with id 2dfc3f7ccf8877dde406b15a8d42f32ff79e2c4fe1235e03218ef60570730b5e Oct 31 00:28:38.137205 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.137178 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:28:38.139054 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.139030 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pghrd" Oct 31 00:28:38.141390 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:38.141266 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e79efa_c2ed_4cc0_928c_e0b9abf5275b.slice/crio-9a7aad1aa8e94fa8a746240dcea56145b51be9f17b836a8a1ec4e3860c9a4e39 WatchSource:0}: Error finding container 9a7aad1aa8e94fa8a746240dcea56145b51be9f17b836a8a1ec4e3860c9a4e39: Status 404 returned error can't find the container with id 9a7aad1aa8e94fa8a746240dcea56145b51be9f17b836a8a1ec4e3860c9a4e39 Oct 31 00:28:38.148817 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:38.148791 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db4e02b_1e7a_4361_994e_f30507dbf0e7.slice/crio-5e3112303462bb8e25b0958bc4904b479a86d527e72103b056d912e1df983c7c WatchSource:0}: Error finding container 5e3112303462bb8e25b0958bc4904b479a86d527e72103b056d912e1df983c7c: Status 404 returned error can't find the container with id 5e3112303462bb8e25b0958bc4904b479a86d527e72103b056d912e1df983c7c Oct 31 00:28:38.151599 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:38.151578 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3073e91a_8363_4d2a_8774_635302d4b040.slice/crio-6585c48b6b476cf57dd29c701e10242c6998dcf16f086da180f099973f1cdc9c WatchSource:0}: Error finding container 6585c48b6b476cf57dd29c701e10242c6998dcf16f086da180f099973f1cdc9c: Status 404 returned error can't find the container with id 6585c48b6b476cf57dd29c701e10242c6998dcf16f086da180f099973f1cdc9c Oct 31 00:28:38.156710 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.156687 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b64jh" Oct 31 00:28:38.158936 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.158894 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:28:38.160553 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.160528 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:28:38.164561 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.164540 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wvff6" Oct 31 00:28:38.169767 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.169745 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:28:38.173478 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:38.173336 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc892baf2_ce51_4c47_8d79_f75f4f50c09d.slice/crio-8306d8cae5a33092875f586bcde7ed1706c565c930d21e4863f49018b186e564 WatchSource:0}: Error finding container 8306d8cae5a33092875f586bcde7ed1706c565c930d21e4863f49018b186e564: Status 404 returned error can't find the container with id 8306d8cae5a33092875f586bcde7ed1706c565c930d21e4863f49018b186e564 Oct 31 00:28:38.174078 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:38.173965 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8478f3_fe5c_4c04_b0a7_08bb736f3ede.slice/crio-dcdf64c6889bf8cc02238d1d6c2baaccc2dbfbe5c9aebaea56077770eca5b327 WatchSource:0}: Error finding container dcdf64c6889bf8cc02238d1d6c2baaccc2dbfbe5c9aebaea56077770eca5b327: Status 404 returned error can't find the container with id dcdf64c6889bf8cc02238d1d6c2baaccc2dbfbe5c9aebaea56077770eca5b327 Oct 31 00:28:38.175351 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:38.175333 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dca6aeb_205b_4634_9216_a98ba3957025.slice/crio-2e23bab50c3ff188c8886ae680ea2ed80410934050277e12433468e15d3bbf07 WatchSource:0}: Error finding container 2e23bab50c3ff188c8886ae680ea2ed80410934050277e12433468e15d3bbf07: Status 404 returned error can't find the container with id 2e23bab50c3ff188c8886ae680ea2ed80410934050277e12433468e15d3bbf07 Oct 31 00:28:38.180723 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:38.180699 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94485d7_b1a8_4caa_bfac_39c7cf29bfaa.slice/crio-cde84019a225b3e17600718d17090867122a504362b61485753b93ef0f2767d9 WatchSource:0}: Error finding container cde84019a225b3e17600718d17090867122a504362b61485753b93ef0f2767d9: Status 404 returned error can't find the container with id cde84019a225b3e17600718d17090867122a504362b61485753b93ef0f2767d9 Oct 31 00:28:38.183012 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:28:38.182987 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b89acd_a759_4036_bd91_16ee9ad9e288.slice/crio-ed0544758ce9298b0c6d9f2ab0d50e637724e41dddb40c4cb5380dec8282de29 WatchSource:0}: Error finding container ed0544758ce9298b0c6d9f2ab0d50e637724e41dddb40c4cb5380dec8282de29: Status 404 returned error can't find the container with id ed0544758ce9298b0c6d9f2ab0d50e637724e41dddb40c4cb5380dec8282de29 Oct 31 00:28:38.422318 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.422282 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs\") pod \"network-metrics-daemon-jcdhx\" (UID: \"ba178af9-0f61-4c7b-a5ab-36e772495016\") " pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:38.422507 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:38.422434 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:28:38.422564 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:38.422515 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs podName:ba178af9-0f61-4c7b-a5ab-36e772495016 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:39.422496702 +0000 UTC m=+3.984456146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs") pod "network-metrics-daemon-jcdhx" (UID: "ba178af9-0f61-4c7b-a5ab-36e772495016") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:28:38.523158 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.523117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:38.523324 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:38.523241 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:28:38.523324 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:38.523304 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret podName:e7b81351-0c71-49b6-bd86-e48483b8b555 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:39.523287185 +0000 UTC m=+4.085246607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret") pod "global-pull-secret-syncer-dvl2d" (UID: "e7b81351-0c71-49b6-bd86-e48483b8b555") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:28:38.623617 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.623577 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lkc\" (UniqueName: \"kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc\") pod \"network-check-target-zwhbr\" (UID: \"6f86ab44-a957-4e07-8bdf-5a8f813f27d4\") " pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:38.623844 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:38.623825 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:28:38.623909 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:38.623852 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:28:38.623909 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:38.623865 2575 projected.go:196] Error preparing data for projected volume kube-api-access-f7lkc for pod openshift-network-diagnostics/network-check-target-zwhbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:28:38.623978 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:38.623920 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc podName:6f86ab44-a957-4e07-8bdf-5a8f813f27d4 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:39.623903406 +0000 UTC m=+4.185862830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7lkc" (UniqueName: "kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc") pod "network-check-target-zwhbr" (UID: "6f86ab44-a957-4e07-8bdf-5a8f813f27d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:28:38.863462 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.863397 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:23:37 +0000 UTC" deadline="2028-06-13 00:12:27.904343157 +0000 UTC" Oct 31 00:28:38.863462 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:38.863454 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14183h43m49.040892988s" Oct 31 00:28:39.003100 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:39.002406 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:39.003100 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:39.002751 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:28:39.013078 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:39.013000 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b64jh" event={"ID":"c892baf2-ce51-4c47-8d79-f75f4f50c09d","Type":"ContainerStarted","Data":"8306d8cae5a33092875f586bcde7ed1706c565c930d21e4863f49018b186e564"} Oct 31 00:28:39.015712 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:39.015621 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pghrd" event={"ID":"3073e91a-8363-4d2a-8774-635302d4b040","Type":"ContainerStarted","Data":"6585c48b6b476cf57dd29c701e10242c6998dcf16f086da180f099973f1cdc9c"} Oct 31 00:28:39.032451 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:39.032309 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" event={"ID":"1db4e02b-1e7a-4361-994e-f30507dbf0e7","Type":"ContainerStarted","Data":"5e3112303462bb8e25b0958bc4904b479a86d527e72103b056d912e1df983c7c"} Oct 31 00:28:39.034601 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:39.034570 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sm6rp" event={"ID":"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0","Type":"ContainerStarted","Data":"2dfc3f7ccf8877dde406b15a8d42f32ff79e2c4fe1235e03218ef60570730b5e"} Oct 31 00:28:39.044900 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:39.044019 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" event={"ID":"2b347d22e86cd58c1b6312f0f07ef259","Type":"ContainerStarted","Data":"15fdbf75cb7433c19267532a7075108f0d65abefd59b5a24912500e29de12c2b"} Oct 31 00:28:39.047956 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:39.047900 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wvff6" event={"ID":"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa","Type":"ContainerStarted","Data":"cde84019a225b3e17600718d17090867122a504362b61485753b93ef0f2767d9"} Oct 31 00:28:39.057946 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:39.057871 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" event={"ID":"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede","Type":"ContainerStarted","Data":"dcdf64c6889bf8cc02238d1d6c2baaccc2dbfbe5c9aebaea56077770eca5b327"} Oct 31 00:28:39.066024 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:39.065877 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" event={"ID":"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b","Type":"ContainerStarted","Data":"9a7aad1aa8e94fa8a746240dcea56145b51be9f17b836a8a1ec4e3860c9a4e39"} Oct 31 00:28:39.071241 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:39.071080 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9l52l" event={"ID":"79b89acd-a759-4036-bd91-16ee9ad9e288","Type":"ContainerStarted","Data":"ed0544758ce9298b0c6d9f2ab0d50e637724e41dddb40c4cb5380dec8282de29"} Oct 31 00:28:39.074182 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:39.074111 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dcljc" event={"ID":"6dca6aeb-205b-4634-9216-a98ba3957025","Type":"ContainerStarted","Data":"2e23bab50c3ff188c8886ae680ea2ed80410934050277e12433468e15d3bbf07"} Oct 31 00:28:39.433849 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:39.433806 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs\") pod \"network-metrics-daemon-jcdhx\" (UID: \"ba178af9-0f61-4c7b-a5ab-36e772495016\") " pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:39.434041 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:39.433953 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:28:39.434041 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:39.434014 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs podName:ba178af9-0f61-4c7b-a5ab-36e772495016 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:41.433995897 +0000 UTC m=+5.995955318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs") pod "network-metrics-daemon-jcdhx" (UID: "ba178af9-0f61-4c7b-a5ab-36e772495016") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:28:39.534198 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:39.534160 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:39.534418 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:39.534401 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:28:39.534503 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:39.534491 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret podName:e7b81351-0c71-49b6-bd86-e48483b8b555 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:41.534455814 +0000 UTC m=+6.096415239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret") pod "global-pull-secret-syncer-dvl2d" (UID: "e7b81351-0c71-49b6-bd86-e48483b8b555") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:28:39.638117 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:39.635545 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lkc\" (UniqueName: \"kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc\") pod \"network-check-target-zwhbr\" (UID: \"6f86ab44-a957-4e07-8bdf-5a8f813f27d4\") " pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:39.638117 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:39.635755 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:28:39.638117 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:39.635776 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:28:39.638117 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:39.635788 2575 projected.go:196] Error preparing data for projected volume kube-api-access-f7lkc for pod openshift-network-diagnostics/network-check-target-zwhbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:28:39.638117 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:39.635844 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc podName:6f86ab44-a957-4e07-8bdf-5a8f813f27d4 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:41.635826539 +0000 UTC m=+6.197785964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7lkc" (UniqueName: "kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc") pod "network-check-target-zwhbr" (UID: "6f86ab44-a957-4e07-8bdf-5a8f813f27d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:28:39.864309 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:39.864268 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:23:37 +0000 UTC" deadline="2028-06-19 15:01:55.890130683 +0000 UTC" Oct 31 00:28:39.864309 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:39.864308 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14342h33m16.02582751s" Oct 31 00:28:40.002650 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:40.002612 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:40.002839 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:40.002775 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:28:40.003103 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:40.003078 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:40.003217 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:40.003200 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:28:40.095687 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:40.093465 2575 generic.go:356] "Generic (PLEG): container finished" podID="bb8ff3cdd197b5db95c89f56a4f8fb1a" containerID="c7e41d6549c55b6bab121a73ba88850f604a59551b1c761ab8ba2b33b3c214a2" exitCode=0 Oct 31 00:28:40.095687 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:40.094719 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" event={"ID":"bb8ff3cdd197b5db95c89f56a4f8fb1a","Type":"ContainerDied","Data":"c7e41d6549c55b6bab121a73ba88850f604a59551b1c761ab8ba2b33b3c214a2"} Oct 31 00:28:40.119012 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:40.116683 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" podStartSLOduration=3.116653163 podStartE2EDuration="3.116653163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:39.07122164 +0000 UTC m=+3.633181084" watchObservedRunningTime="2026-10-31 00:28:40.116653163 +0000 UTC m=+4.678612607" Oct 31 00:28:41.002698 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:41.002615 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:41.003126 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:41.002765 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:28:41.104619 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:41.104575 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" event={"ID":"bb8ff3cdd197b5db95c89f56a4f8fb1a","Type":"ContainerStarted","Data":"5f66a2620f261d20d7f3221521773a3b9b12def7aea77138a52b51d04540a24d"} Oct 31 00:28:41.460075 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:41.459784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs\") pod \"network-metrics-daemon-jcdhx\" (UID: \"ba178af9-0f61-4c7b-a5ab-36e772495016\") " pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:41.460075 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:41.459983 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:28:41.460075 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:41.460042 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs podName:ba178af9-0f61-4c7b-a5ab-36e772495016 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:45.460024568 +0000 UTC m=+10.021983995 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs") pod "network-metrics-daemon-jcdhx" (UID: "ba178af9-0f61-4c7b-a5ab-36e772495016") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:28:41.560262 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:41.560214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:41.560430 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:41.560369 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:28:41.560503 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:41.560448 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret podName:e7b81351-0c71-49b6-bd86-e48483b8b555 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:45.560430868 +0000 UTC m=+10.122390294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret") pod "global-pull-secret-syncer-dvl2d" (UID: "e7b81351-0c71-49b6-bd86-e48483b8b555") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:28:41.661131 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:41.661089 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lkc\" (UniqueName: \"kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc\") pod \"network-check-target-zwhbr\" (UID: \"6f86ab44-a957-4e07-8bdf-5a8f813f27d4\") " pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:41.661346 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:41.661325 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:28:41.661432 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:41.661351 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:28:41.661432 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:41.661364 2575 projected.go:196] Error preparing data for projected volume kube-api-access-f7lkc for pod openshift-network-diagnostics/network-check-target-zwhbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:28:41.661432 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:41.661421 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc podName:6f86ab44-a957-4e07-8bdf-5a8f813f27d4 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:45.661403159 +0000 UTC m=+10.223362600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7lkc" (UniqueName: "kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc") pod "network-check-target-zwhbr" (UID: "6f86ab44-a957-4e07-8bdf-5a8f813f27d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:28:42.004264 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:42.004229 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:42.004749 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:42.004361 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:28:42.004749 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:42.004438 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:42.004749 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:42.004511 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:28:42.113712 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:42.113654 2575 generic.go:356] "Generic (PLEG): container finished" podID="6dca6aeb-205b-4634-9216-a98ba3957025" containerID="8ce55a93270d2f9f8f1ad8bc617a29813da3fc0a132c2a9a9d693cb9012f57eb" exitCode=0 Oct 31 00:28:42.114779 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:42.114737 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dcljc" event={"ID":"6dca6aeb-205b-4634-9216-a98ba3957025","Type":"ContainerDied","Data":"8ce55a93270d2f9f8f1ad8bc617a29813da3fc0a132c2a9a9d693cb9012f57eb"} Oct 31 00:28:42.146467 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:42.146401 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" podStartSLOduration=5.146382071 podStartE2EDuration="5.146382071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:41.127003771 +0000 UTC m=+5.688963212" watchObservedRunningTime="2026-10-31 00:28:42.146382071 +0000 UTC m=+6.708341514" Oct 31 00:28:43.001836 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:43.001794 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:43.002008 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:43.001950 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:28:43.122295 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:43.121598 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b64jh" event={"ID":"c892baf2-ce51-4c47-8d79-f75f4f50c09d","Type":"ContainerStarted","Data":"dd5d981c518705a6cac3f947b18a5f14af3113e864b16e163bf0059f1b2b7a60"} Oct 31 00:28:43.805657 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:43.805615 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b64jh" Oct 31 00:28:44.001847 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:44.001821 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:44.001976 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:44.001946 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:28:44.002327 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:44.002308 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:44.002425 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:44.002404 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:28:44.129075 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:44.129047 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9l52l" event={"ID":"79b89acd-a759-4036-bd91-16ee9ad9e288","Type":"ContainerStarted","Data":"f133451f4cefab5b922718b4b6455a51a159383fe5db900a1c7f25dc5ca76ce5"} Oct 31 00:28:44.152705 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:44.152447 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b64jh" podStartSLOduration=4.285002004 podStartE2EDuration="8.152427284s" totalImagesPullingTime="3.86742528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:43.143919873 +0000 UTC m=+7.705879335" watchObservedRunningTime="2026-10-31 00:28:44.152427284 +0000 UTC m=+8.714386728" Oct 31 00:28:44.153527 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:44.153480 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9l52l" podStartSLOduration=2.97807908 podStartE2EDuration="8.153469811s" totalImagesPullingTime="5.175390731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:44.153147888 +0000 UTC m=+8.715107331" watchObservedRunningTime="2026-10-31 00:28:44.153469811 +0000 UTC m=+8.715429254" Oct 31 00:28:44.848313 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:44.848233 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b64jh" Oct 31 00:28:44.848831 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:44.848807 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b64jh" Oct 31 00:28:45.002016 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:45.001987 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:45.002202 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:45.002120 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:28:45.133470 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:45.132912 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" event={"ID":"1db4e02b-1e7a-4361-994e-f30507dbf0e7","Type":"ContainerStarted","Data":"9fb95b543b39c42faed13ff68e2f7b86360579ed418fdc4ae106890fc1d7ef49"} Oct 31 00:28:45.134373 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:45.134156 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b64jh" Oct 31 00:28:45.492636 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:45.492554 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs\") pod \"network-metrics-daemon-jcdhx\" (UID: \"ba178af9-0f61-4c7b-a5ab-36e772495016\") " pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:45.492832 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:45.492730 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:28:45.492832 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:45.492804 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs podName:ba178af9-0f61-4c7b-a5ab-36e772495016 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:53.492783278 +0000 UTC m=+18.054742700 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs") pod "network-metrics-daemon-jcdhx" (UID: "ba178af9-0f61-4c7b-a5ab-36e772495016") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:28:45.594055 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:45.593538 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:45.594055 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:45.593693 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:28:45.594055 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:45.593750 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret podName:e7b81351-0c71-49b6-bd86-e48483b8b555 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:53.593734549 +0000 UTC m=+18.155693987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret") pod "global-pull-secret-syncer-dvl2d" (UID: "e7b81351-0c71-49b6-bd86-e48483b8b555") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:28:45.694569 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:45.693859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lkc\" (UniqueName: \"kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc\") pod \"network-check-target-zwhbr\" (UID: \"6f86ab44-a957-4e07-8bdf-5a8f813f27d4\") " pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:45.694569 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:45.694072 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:28:45.694569 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:45.694095 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:28:45.694569 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:45.694107 2575 projected.go:196] Error preparing data for projected volume kube-api-access-f7lkc for pod openshift-network-diagnostics/network-check-target-zwhbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:28:45.694569 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:45.694161 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc podName:6f86ab44-a957-4e07-8bdf-5a8f813f27d4 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:53.694145168 +0000 UTC m=+18.256104592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7lkc" (UniqueName: "kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc") pod "network-check-target-zwhbr" (UID: "6f86ab44-a957-4e07-8bdf-5a8f813f27d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:28:45.817641 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:45.817604 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:28:45.872954 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:45.872235 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:28:45.817623224Z","UUID":"34fd74a3-1b2c-4a14-ade3-c8bdfcc5ab00","Handler":null,"Name":"","Endpoint":""} Oct 31 00:28:45.877921 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:45.877423 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:28:45.877921 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:45.877443 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:28:46.008942 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:46.008361 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:46.008942 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:46.008397 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:46.008942 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:46.008479 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:28:46.008942 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:46.008901 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:28:46.136095 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:46.136065 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wvff6" event={"ID":"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa","Type":"ContainerStarted","Data":"5387b3b761ab7511bd32423a4bcc9cb53d889801c4f00f1441f523f3809bd1d2"} Oct 31 00:28:46.139312 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:46.139290 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" event={"ID":"1db4e02b-1e7a-4361-994e-f30507dbf0e7","Type":"ContainerStarted","Data":"24d92bea04b6bb728e6341107c1793c567fa4493fd6162db669fb3cf59040f8a"} Oct 31 00:28:47.002038 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:47.001806 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:47.002181 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:47.002149 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:28:47.144117 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:47.144077 2575 generic.go:356] "Generic (PLEG): container finished" podID="6dca6aeb-205b-4634-9216-a98ba3957025" containerID="a98f081c5e926131f2498738aaea20a4b13790a56f07a2229b89a9b1a7f767ce" exitCode=0 Oct 31 00:28:47.144714 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:47.144183 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dcljc" event={"ID":"6dca6aeb-205b-4634-9216-a98ba3957025","Type":"ContainerDied","Data":"a98f081c5e926131f2498738aaea20a4b13790a56f07a2229b89a9b1a7f767ce"} Oct 31 00:28:47.146930 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:47.146903 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pghrd" event={"ID":"3073e91a-8363-4d2a-8774-635302d4b040","Type":"ContainerStarted","Data":"4c08fbecf9ba200554ac0c7cd3aaa40e8e853b7448959c1ccfb2ac5fa949f4cc"} Oct 31 00:28:47.150021 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:47.149987 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" event={"ID":"1db4e02b-1e7a-4361-994e-f30507dbf0e7","Type":"ContainerStarted","Data":"789d82e853a81e3a90c39a6b982c282ee9a3160dc7a2b1cce4065fc8fe927809"} Oct 31 00:28:47.185608 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:47.185519 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-wvff6" podStartSLOduration=3.503412181 podStartE2EDuration="11.185500477s" totalImagesPullingTime="7.682088296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:46.164727912 +0000 UTC m=+10.726687355" watchObservedRunningTime="2026-10-31 00:28:47.185500477 +0000 UTC m=+11.747459919" Oct 31 00:28:47.209243 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:47.209191 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-pghrd" podStartSLOduration=3.294217044 podStartE2EDuration="11.209173398s" totalImagesPullingTime="7.914956354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:47.208549461 +0000 UTC m=+11.770508904" watchObservedRunningTime="2026-10-31 00:28:47.209173398 +0000 UTC m=+11.771132841" Oct 31 00:28:47.242812 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:47.242768 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" podStartSLOduration=2.847667516 podStartE2EDuration="11.24275223s" totalImagesPullingTime="8.395084714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:47.242132605 +0000 UTC m=+11.804092047" watchObservedRunningTime="2026-10-31 00:28:47.24275223 +0000 UTC m=+11.804711675" Oct 31 00:28:48.006270 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:48.006197 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:48.006270 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:48.006235 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:48.006465 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:48.006314 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:28:48.006465 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:48.006438 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:28:48.153387 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:48.153316 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sm6rp" event={"ID":"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0","Type":"ContainerStarted","Data":"0e0e318ae9e1e52e448852fc3d54843cb2cef07c7ded67bb50f2eb24049f1a28"} Oct 31 00:28:48.175431 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:48.175390 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sm6rp" podStartSLOduration=4.100276579 podStartE2EDuration="12.175354312s" totalImagesPullingTime="8.075077733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:48.175109864 +0000 UTC m=+12.737069305" watchObservedRunningTime="2026-10-31 00:28:48.175354312 +0000 UTC m=+12.737313754" Oct 31 00:28:49.002245 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:49.002212 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:49.002416 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:49.002351 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:28:49.157443 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:49.157391 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" event={"ID":"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede","Type":"ContainerStarted","Data":"d1e9b4a895ee7a853a3db54d9f28d9d2804292c2d15dd24847bbc84c0862e130"} Oct 31 00:28:49.159836 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:49.159808 2575 generic.go:356] "Generic (PLEG): container finished" podID="6dca6aeb-205b-4634-9216-a98ba3957025" containerID="3af11775f85dfdcaaeae708b1732561ab92b7ddcac63ddcd905d3c5d2b274177" exitCode=0 Oct 31 00:28:49.159965 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:49.159899 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dcljc" event={"ID":"6dca6aeb-205b-4634-9216-a98ba3957025","Type":"ContainerDied","Data":"3af11775f85dfdcaaeae708b1732561ab92b7ddcac63ddcd905d3c5d2b274177"} Oct 31 00:28:49.182020 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:49.181978 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" podStartSLOduration=2.773602486 podStartE2EDuration="13.181965221s" totalImagesPullingTime="10.408362735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:49.181496747 +0000 UTC m=+13.743456184" watchObservedRunningTime="2026-10-31 00:28:49.181965221 +0000 UTC m=+13.743924662" Oct 31 00:28:50.001826 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:50.001795 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:50.001826 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:50.001834 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:50.002001 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:50.001907 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:28:50.002075 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:50.002046 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:28:50.165071 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:50.164826 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dcljc" event={"ID":"6dca6aeb-205b-4634-9216-a98ba3957025","Type":"ContainerStarted","Data":"59a8d5ac93a28fada17d7d14a8c8b947ebafa5602198f214f31813b8170d4665"} Oct 31 00:28:51.001730 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:51.001690 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:51.001924 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:51.001815 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:28:51.169929 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:51.169892 2575 generic.go:356] "Generic (PLEG): container finished" podID="6dca6aeb-205b-4634-9216-a98ba3957025" containerID="59a8d5ac93a28fada17d7d14a8c8b947ebafa5602198f214f31813b8170d4665" exitCode=0 Oct 31 00:28:51.170356 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:51.169943 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dcljc" event={"ID":"6dca6aeb-205b-4634-9216-a98ba3957025","Type":"ContainerDied","Data":"59a8d5ac93a28fada17d7d14a8c8b947ebafa5602198f214f31813b8170d4665"} Oct 31 00:28:52.002736 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:52.002698 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:52.002890 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:52.002732 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:52.002890 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:52.002834 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:28:52.003008 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:52.002928 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:28:53.002508 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:53.002326 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:53.003175 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:53.002612 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:28:53.176860 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:53.176798 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:28:53.177126 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:53.177107 2575 generic.go:356] "Generic (PLEG): container finished" podID="e7e79efa-c2ed-4cc0-928c-e0b9abf5275b" containerID="ae6d5c1751eeff5b784efbc14231c8b255feba4c2ed10b76dedb026486e5f0a3" exitCode=1 Oct 31 00:28:53.177180 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:53.177162 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" event={"ID":"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b","Type":"ContainerStarted","Data":"aadfe6d15a655e4be99a342fa3d85207c28b2092172cdc758430d2eeaa12d0cc"} Oct 31 00:28:53.177232 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:53.177221 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" event={"ID":"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b","Type":"ContainerStarted","Data":"cda45546d7043d2dbf5efb465285ad9086c66ffab0e7d771e5412b2528e90d5b"} Oct 31 00:28:53.177278 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:53.177266 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" event={"ID":"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b","Type":"ContainerStarted","Data":"181fa2f18235cae49f71b1d3658d86a42f90015ce94f6ff30b959395961451dc"} Oct 31 00:28:53.177321 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:53.177308 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" event={"ID":"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b","Type":"ContainerStarted","Data":"c0d7fad8dc7cb6055c37f864e6a16ef2baae9abbc555e9ef68844fa46b82b21f"} Oct 31 00:28:53.177355 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:53.177346 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" event={"ID":"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b","Type":"ContainerDied","Data":"ae6d5c1751eeff5b784efbc14231c8b255feba4c2ed10b76dedb026486e5f0a3"} Oct 31 00:28:53.177399 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:53.177386 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" event={"ID":"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b","Type":"ContainerStarted","Data":"8240c438df8ca434f7b8f189f7df0849c6a5cb168f960ab5b46611f078e595ad"} Oct 31 00:28:53.179743 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:53.179725 2575 generic.go:356] "Generic (PLEG): container finished" podID="6dca6aeb-205b-4634-9216-a98ba3957025" containerID="d61d0a845291a0f2cf37c076996db842683dc842f4a5f12f18a247a6bb8e24b0" exitCode=0 Oct 31 00:28:53.179816 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:53.179765 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dcljc" event={"ID":"6dca6aeb-205b-4634-9216-a98ba3957025","Type":"ContainerDied","Data":"d61d0a845291a0f2cf37c076996db842683dc842f4a5f12f18a247a6bb8e24b0"} Oct 31 00:28:53.552088 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:53.552020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs\") pod \"network-metrics-daemon-jcdhx\" (UID: \"ba178af9-0f61-4c7b-a5ab-36e772495016\") " pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:53.552223 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:53.552126 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:28:53.552223 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:53.552173 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs podName:ba178af9-0f61-4c7b-a5ab-36e772495016 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:09.552161628 +0000 UTC m=+34.114121048 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs") pod "network-metrics-daemon-jcdhx" (UID: "ba178af9-0f61-4c7b-a5ab-36e772495016") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:28:53.653018 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:53.652988 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:53.653165 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:53.653083 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:28:53.653165 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:53.653135 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret podName:e7b81351-0c71-49b6-bd86-e48483b8b555 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:09.653123511 +0000 UTC m=+34.215082944 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret") pod "global-pull-secret-syncer-dvl2d" (UID: "e7b81351-0c71-49b6-bd86-e48483b8b555") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:28:53.753259 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:53.753225 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lkc\" (UniqueName: \"kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc\") pod \"network-check-target-zwhbr\" (UID: \"6f86ab44-a957-4e07-8bdf-5a8f813f27d4\") " pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:53.753400 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:53.753368 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:28:53.753400 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:53.753389 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:28:53.753400 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:53.753399 2575 projected.go:196] Error preparing data for projected volume kube-api-access-f7lkc for pod openshift-network-diagnostics/network-check-target-zwhbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:28:53.753497 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:53.753448 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc podName:6f86ab44-a957-4e07-8bdf-5a8f813f27d4 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:09.753436052 +0000 UTC m=+34.315395473 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7lkc" (UniqueName: "kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc") pod "network-check-target-zwhbr" (UID: "6f86ab44-a957-4e07-8bdf-5a8f813f27d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:28:54.002327 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:54.002295 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:54.002327 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:54.002295 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:54.002563 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:54.002451 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:28:54.002563 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:54.002475 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:28:54.184088 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:54.184056 2575 generic.go:356] "Generic (PLEG): container finished" podID="6dca6aeb-205b-4634-9216-a98ba3957025" containerID="3c2ab6f88e9e540056a0ab52280e2b1c8760c41f2669da8a50f00c35c36a6431" exitCode=0 Oct 31 00:28:54.184244 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:54.184111 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dcljc" event={"ID":"6dca6aeb-205b-4634-9216-a98ba3957025","Type":"ContainerDied","Data":"3c2ab6f88e9e540056a0ab52280e2b1c8760c41f2669da8a50f00c35c36a6431"} Oct 31 00:28:55.002341 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:55.002265 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:55.002475 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:55.002365 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:28:55.188312 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:55.188285 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:28:55.188819 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:55.188608 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" event={"ID":"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b","Type":"ContainerStarted","Data":"575fc8fc40e60c03a9415eb6e698b10a608f5def7aff739da7a12c4f14d323a1"} Oct 31 00:28:55.191712 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:55.191688 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dcljc" event={"ID":"6dca6aeb-205b-4634-9216-a98ba3957025","Type":"ContainerStarted","Data":"381255f4ca8e07e68a75b07b2d748b929005b17d16208f9dc929682af8b53878"} Oct 31 00:28:56.004233 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:56.004204 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:56.004391 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:56.004331 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:28:56.004391 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:56.004387 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:56.004469 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:56.004449 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:28:56.791444 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:56.791273 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wvff6_f94485d7-b1a8-4caa-bfac-39c7cf29bfaa/dns-node-resolver/0.log" Oct 31 00:28:57.002389 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:57.002292 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:57.002548 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:57.002401 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:28:57.038049 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:57.038019 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b64jh_c892baf2-ce51-4c47-8d79-f75f4f50c09d/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:57.199201 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:57.199177 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:28:57.199509 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:57.199490 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" event={"ID":"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b","Type":"ContainerStarted","Data":"956ede38362ce89a574d8cae9baf58cd445d0988bbb2d747d67c20e62df1addb"} Oct 31 00:28:57.199815 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:57.199795 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:57.199915 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:57.199874 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:57.199972 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:57.199922 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:57.200016 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:57.199989 2575 scope.go:122] "RemoveContainer" containerID="ae6d5c1751eeff5b784efbc14231c8b255feba4c2ed10b76dedb026486e5f0a3" Oct 31 00:28:57.215712 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:57.215691 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:57.216362 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:57.216346 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:28:57.238235 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:57.238190 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dcljc" podStartSLOduration=10.356027816 podStartE2EDuration="21.238176948s" totalImagesPullingTime="10.882149132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:55.222820702 +0000 UTC m=+19.784780144" watchObservedRunningTime="2026-10-31 00:28:57.238176948 +0000 UTC m=+21.800136381" Oct 31 00:28:57.340491 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:57.340424 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b64jh_c892baf2-ce51-4c47-8d79-f75f4f50c09d/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:57.556549 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:57.556527 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9l52l_79b89acd-a759-4036-bd91-16ee9ad9e288/node-ca/0.log" Oct 31 00:28:57.698786 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:57.698756 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b64jh_c892baf2-ce51-4c47-8d79-f75f4f50c09d/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:57.921774 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:57.921748 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b64jh_c892baf2-ce51-4c47-8d79-f75f4f50c09d/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:58.005082 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:58.005023 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:28:58.005189 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:58.005023 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:28:58.005189 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:58.005118 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:28:58.005189 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:58.005177 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:28:58.100879 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:58.100854 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b64jh_c892baf2-ce51-4c47-8d79-f75f4f50c09d/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:58.205222 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:58.205199 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:28:58.205567 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:58.205549 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" event={"ID":"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b","Type":"ContainerStarted","Data":"66e2cf08729a3833eca1114f28c95dd77b3bae65aaad1ae1363d3484ac66ac36"} Oct 31 00:28:58.247459 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:58.247395 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" podStartSLOduration=9.273474477 podStartE2EDuration="23.247379048s" totalImagesPullingTime="13.973904571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:58.246594277 +0000 UTC m=+22.808553701" watchObservedRunningTime="2026-10-31 00:28:58.247379048 +0000 UTC m=+22.809338492" Oct 31 00:28:59.002362 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:28:59.002334 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:28:59.002763 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:28:59.002440 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:29:00.004821 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:00.004793 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:00.005361 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:00.004915 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:29:00.005361 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:00.004956 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:00.005361 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:00.005021 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:29:01.001899 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:01.001861 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:29:01.002094 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:01.001989 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:29:02.001700 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:02.001659 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:02.002101 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:02.001763 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:29:02.002101 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:02.001819 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:02.002101 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:02.001905 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:29:03.002124 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:03.002095 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:29:03.002485 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:03.002231 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:29:04.002341 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:04.002310 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:04.002341 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:04.002335 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:04.002890 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:04.002417 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:29:04.002890 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:04.002520 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:29:05.002039 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:05.001997 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:29:05.002230 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:05.002138 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:29:06.004309 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:06.004285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:06.004691 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:06.004372 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:29:06.004691 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:06.004443 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:06.004691 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:06.004528 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:29:07.001838 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:07.001807 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:29:07.002002 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:07.001906 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:29:08.002163 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:08.002133 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:08.002621 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:08.002235 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:29:08.002621 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:08.002281 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:08.002621 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:08.002341 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:29:09.001818 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:09.001788 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:29:09.002092 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:09.001909 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:29:09.569752 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:09.569722 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs\") pod \"network-metrics-daemon-jcdhx\" (UID: \"ba178af9-0f61-4c7b-a5ab-36e772495016\") " pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:29:09.570134 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:09.569832 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:29:09.570134 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:09.569886 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs podName:ba178af9-0f61-4c7b-a5ab-36e772495016 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:41.569869832 +0000 UTC m=+66.131829266 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs") pod "network-metrics-daemon-jcdhx" (UID: "ba178af9-0f61-4c7b-a5ab-36e772495016") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:29:09.670328 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:09.670294 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:09.670474 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:09.670429 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:29:09.670512 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:09.670487 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret podName:e7b81351-0c71-49b6-bd86-e48483b8b555 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:41.670473403 +0000 UTC m=+66.232432824 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret") pod "global-pull-secret-syncer-dvl2d" (UID: "e7b81351-0c71-49b6-bd86-e48483b8b555") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:29:09.770768 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:09.770742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lkc\" (UniqueName: \"kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc\") pod \"network-check-target-zwhbr\" (UID: \"6f86ab44-a957-4e07-8bdf-5a8f813f27d4\") " pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:09.770921 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:09.770867 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:29:09.770921 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:09.770880 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:29:09.770921 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:09.770890 2575 projected.go:196] Error preparing data for projected volume kube-api-access-f7lkc for pod openshift-network-diagnostics/network-check-target-zwhbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:29:09.771023 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:09.770934 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc podName:6f86ab44-a957-4e07-8bdf-5a8f813f27d4 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:41.770919948 +0000 UTC m=+66.332879368 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7lkc" (UniqueName: "kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc") pod "network-check-target-zwhbr" (UID: "6f86ab44-a957-4e07-8bdf-5a8f813f27d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:29:10.002531 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:10.002499 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:10.002694 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:10.002505 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:10.002694 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:10.002619 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:29:10.002776 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:10.002696 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:29:11.001812 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:11.001782 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:29:11.002468 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:11.002437 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:29:11.680934 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:11.680629 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dvl2d"] Oct 31 00:29:11.680934 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:11.680922 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zwhbr"] Oct 31 00:29:11.681174 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:11.680995 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:11.681174 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:11.681099 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:29:11.681174 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:11.681164 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:11.681325 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:11.681244 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:29:11.682374 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:11.682349 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jcdhx"] Oct 31 00:29:11.682512 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:11.682448 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:29:11.682572 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:11.682548 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:29:13.002466 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:13.002435 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:13.002942 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:13.002530 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:29:14.002522 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:14.002489 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:14.002903 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:14.002594 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dvl2d" podUID="e7b81351-0c71-49b6-bd86-e48483b8b555" Oct 31 00:29:14.002903 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:14.002626 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:29:14.002903 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:14.002707 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcdhx" podUID="ba178af9-0f61-4c7b-a5ab-36e772495016" Oct 31 00:29:15.001660 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.001630 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:15.001838 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:15.001741 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhbr" podUID="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" Oct 31 00:29:15.773164 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.773135 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeReady" Oct 31 00:29:15.773573 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.773246 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 00:29:15.835328 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.835300 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xkqrm"] Oct 31 00:29:15.856923 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.856903 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pnmtp"] Oct 31 00:29:15.857055 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.857036 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:15.868681 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.868635 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:29:15.869218 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.869202 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:29:15.869218 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.869203 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8j7n8\"" Oct 31 00:29:15.872942 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.872917 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xkqrm"] Oct 31 00:29:15.873041 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.872963 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pnmtp"] Oct 31 00:29:15.873041 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.872986 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w49pc"] Oct 31 00:29:15.873041 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.872998 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pnmtp" Oct 31 00:29:15.887196 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.887172 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w49pc"] Oct 31 00:29:15.887349 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.887322 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:15.969468 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.969442 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:29:15.969629 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.969493 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:29:15.969811 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.969780 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:29:15.969811 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.969794 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:29:15.969811 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.969795 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nchkl\"" Oct 31 00:29:15.969936 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.969833 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:29:15.969936 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.969866 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wfrwt\"" Oct 31 00:29:15.969936 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.969864 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:29:15.969936 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:15.969914 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:29:16.005033 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.004975 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:16.005147 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.005098 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:29:16.012207 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.012186 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-74f876fbf6-4mlmt"] Oct 31 00:29:16.014186 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.014166 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/37dd17e5-ba62-4f95-bfc7-05b132104e44-config-volume\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:16.014281 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.014202 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57decfb3-58bb-4b9c-b834-205594071bdf-cert\") pod \"ingress-canary-pnmtp\" (UID: \"57decfb3-58bb-4b9c-b834-205594071bdf\") " pod="openshift-ingress-canary/ingress-canary-pnmtp" Oct 31 00:29:16.014281 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.014232 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51dbc54b-218b-4274-9ebb-4cdbab400d22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:16.014281 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.014272 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6gnm\" (UniqueName: \"kubernetes.io/projected/51dbc54b-218b-4274-9ebb-4cdbab400d22-kube-api-access-h6gnm\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:16.014442 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.014426 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6j5l\" (UniqueName: \"kubernetes.io/projected/57decfb3-58bb-4b9c-b834-205594071bdf-kube-api-access-v6j5l\") pod \"ingress-canary-pnmtp\" (UID: \"57decfb3-58bb-4b9c-b834-205594071bdf\") " pod="openshift-ingress-canary/ingress-canary-pnmtp" Oct 31 00:29:16.014505 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.014461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37dd17e5-ba62-4f95-bfc7-05b132104e44-tmp-dir\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:16.014505 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.014489 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvdnb\" (UniqueName: \"kubernetes.io/projected/37dd17e5-ba62-4f95-bfc7-05b132104e44-kube-api-access-wvdnb\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:16.014590 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.014560 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51dbc54b-218b-4274-9ebb-4cdbab400d22-data-volume\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:16.014624 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.014613 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37dd17e5-ba62-4f95-bfc7-05b132104e44-metrics-tls\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:16.014658 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.014647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51dbc54b-218b-4274-9ebb-4cdbab400d22-crio-socket\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:16.014716 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.014662 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51dbc54b-218b-4274-9ebb-4cdbab400d22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:16.035446 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.035423 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74f876fbf6-4mlmt"] Oct 31 00:29:16.035544 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.035506 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.069761 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.069740 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 00:29:16.069897 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.069807 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 00:29:16.069897 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.069871 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 00:29:16.069897 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.069872 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 00:29:16.070036 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.069979 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 00:29:16.070036 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.070002 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 00:29:16.071206 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.071193 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:29:16.071472 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.071456 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4bqfb\"" Oct 31 00:29:16.071548 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.071535 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wfp74\"" Oct 31 00:29:16.071708 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.071694 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hxhg5\"" Oct 31 00:29:16.073585 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.073571 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:29:16.073728 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.073618 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 00:29:16.075334 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.075315 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 00:29:16.115027 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.114994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37dd17e5-ba62-4f95-bfc7-05b132104e44-metrics-tls\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:16.115158 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.115055 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51dbc54b-218b-4274-9ebb-4cdbab400d22-crio-socket\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:16.115158 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.115080 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51dbc54b-218b-4274-9ebb-4cdbab400d22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:16.115158 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.115099 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/37dd17e5-ba62-4f95-bfc7-05b132104e44-config-volume\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:16.115158 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.115134 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57decfb3-58bb-4b9c-b834-205594071bdf-cert\") pod \"ingress-canary-pnmtp\" (UID: \"57decfb3-58bb-4b9c-b834-205594071bdf\") " pod="openshift-ingress-canary/ingress-canary-pnmtp" Oct 31 00:29:16.115334 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.115170 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51dbc54b-218b-4274-9ebb-4cdbab400d22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:16.115334 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.115193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6gnm\" (UniqueName: \"kubernetes.io/projected/51dbc54b-218b-4274-9ebb-4cdbab400d22-kube-api-access-h6gnm\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:16.115334 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.115226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6j5l\" (UniqueName: \"kubernetes.io/projected/57decfb3-58bb-4b9c-b834-205594071bdf-kube-api-access-v6j5l\") pod \"ingress-canary-pnmtp\" (UID: \"57decfb3-58bb-4b9c-b834-205594071bdf\") " pod="openshift-ingress-canary/ingress-canary-pnmtp" Oct 31 00:29:16.115334 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.115248 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37dd17e5-ba62-4f95-bfc7-05b132104e44-tmp-dir\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:16.115334 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.115267 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvdnb\" (UniqueName: \"kubernetes.io/projected/37dd17e5-ba62-4f95-bfc7-05b132104e44-kube-api-access-wvdnb\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:16.115334 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.115301 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51dbc54b-218b-4274-9ebb-4cdbab400d22-crio-socket\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:16.115334 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.115317 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51dbc54b-218b-4274-9ebb-4cdbab400d22-data-volume\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:16.115749 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.115722 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51dbc54b-218b-4274-9ebb-4cdbab400d22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:16.115835 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.115787 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/37dd17e5-ba62-4f95-bfc7-05b132104e44-config-volume\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:16.116096 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.116073 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37dd17e5-ba62-4f95-bfc7-05b132104e44-tmp-dir\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:16.116268 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.116251 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51dbc54b-218b-4274-9ebb-4cdbab400d22-data-volume\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:16.118977 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.118954 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37dd17e5-ba62-4f95-bfc7-05b132104e44-metrics-tls\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:16.119081 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.119023 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51dbc54b-218b-4274-9ebb-4cdbab400d22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:16.119203 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.119184 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57decfb3-58bb-4b9c-b834-205594071bdf-cert\") pod \"ingress-canary-pnmtp\" (UID: \"57decfb3-58bb-4b9c-b834-205594071bdf\") " pod="openshift-ingress-canary/ingress-canary-pnmtp" Oct 31 00:29:16.130144 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.130118 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6gnm\" (UniqueName: \"kubernetes.io/projected/51dbc54b-218b-4274-9ebb-4cdbab400d22-kube-api-access-h6gnm\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:16.130251 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.130124 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvdnb\" (UniqueName: \"kubernetes.io/projected/37dd17e5-ba62-4f95-bfc7-05b132104e44-kube-api-access-wvdnb\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:16.130396 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.130379 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6j5l\" (UniqueName: \"kubernetes.io/projected/57decfb3-58bb-4b9c-b834-205594071bdf-kube-api-access-v6j5l\") pod \"ingress-canary-pnmtp\" (UID: \"57decfb3-58bb-4b9c-b834-205594071bdf\") " pod="openshift-ingress-canary/ingress-canary-pnmtp" Oct 31 00:29:16.193527 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.193494 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:16.216204 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.216184 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pnmtp" Oct 31 00:29:16.216334 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.216285 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mfhm\" (UniqueName: \"kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.216334 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.216314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.216401 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.216344 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91208431-95bb-4e24-ab4b-e0f80f0d6207-tmp\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.216487 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.216466 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.216521 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.216494 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.216554 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.216521 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.216623 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.216609 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.216684 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.216658 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.268348 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.268257 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:16.317849 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.317819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.317994 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.317854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mfhm\" (UniqueName: \"kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.317994 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.317971 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.318101 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.318039 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91208431-95bb-4e24-ab4b-e0f80f0d6207-tmp\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.318101 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.318090 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.318182 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.318113 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.318182 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.318155 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.318280 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.318193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.318392 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.318373 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.319036 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.318633 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91208431-95bb-4e24-ab4b-e0f80f0d6207-tmp\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.319036 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.319008 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.319215 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.319117 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.319215 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.319167 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.321170 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.320829 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.321949 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.321928 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.338095 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.337401 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mfhm\" (UniqueName: \"kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.398485 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.398450 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:16.578706 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.578595 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pnmtp"] Oct 31 00:29:16.579327 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.579254 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xkqrm"] Oct 31 00:29:16.580028 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.579954 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w49pc"] Oct 31 00:29:16.583924 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:29:16.583884 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37dd17e5_ba62_4f95_bfc7_05b132104e44.slice/crio-eb1837caaa867184184cb737ed2ca7fbe06a6b484638b74d5fd579b1b5b896fd WatchSource:0}: Error finding container eb1837caaa867184184cb737ed2ca7fbe06a6b484638b74d5fd579b1b5b896fd: Status 404 returned error can't find the container with id eb1837caaa867184184cb737ed2ca7fbe06a6b484638b74d5fd579b1b5b896fd Oct 31 00:29:16.584513 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:29:16.584493 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57decfb3_58bb_4b9c_b834_205594071bdf.slice/crio-290dfca05c359ae7d59ff7029b87f39951a55146fa46cfff9478e29183348f0e WatchSource:0}: Error finding container 290dfca05c359ae7d59ff7029b87f39951a55146fa46cfff9478e29183348f0e: Status 404 returned error can't find the container with id 290dfca05c359ae7d59ff7029b87f39951a55146fa46cfff9478e29183348f0e Oct 31 00:29:16.585477 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:29:16.585446 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51dbc54b_218b_4274_9ebb_4cdbab400d22.slice/crio-1f0fb6a0696bb35aba27a48e5b95aea97af5c07216c282ab7cd1925e39c6408c WatchSource:0}: Error finding container 1f0fb6a0696bb35aba27a48e5b95aea97af5c07216c282ab7cd1925e39c6408c: Status 404 returned error can't find the container with id 1f0fb6a0696bb35aba27a48e5b95aea97af5c07216c282ab7cd1925e39c6408c Oct 31 00:29:16.691483 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:16.691414 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74f876fbf6-4mlmt"] Oct 31 00:29:16.695649 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:29:16.695624 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91208431_95bb_4e24_ab4b_e0f80f0d6207.slice/crio-91572e8ab1e6067a5d2f4f9a1e81b1859c2d37d3048dda1a2a4aa345af7b7b2a WatchSource:0}: Error finding container 91572e8ab1e6067a5d2f4f9a1e81b1859c2d37d3048dda1a2a4aa345af7b7b2a: Status 404 returned error can't find the container with id 91572e8ab1e6067a5d2f4f9a1e81b1859c2d37d3048dda1a2a4aa345af7b7b2a Oct 31 00:29:17.002131 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:17.002101 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:17.008453 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:17.008270 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:29:17.008453 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:17.008289 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qx27s\"" Oct 31 00:29:17.008453 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:17.008342 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:29:17.247297 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:17.247263 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w49pc" event={"ID":"51dbc54b-218b-4274-9ebb-4cdbab400d22","Type":"ContainerStarted","Data":"016f916693f5d882b9275d338b565b722fb594c2c6f55f2c77eeda5b941f3e44"} Oct 31 00:29:17.247458 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:17.247332 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w49pc" event={"ID":"51dbc54b-218b-4274-9ebb-4cdbab400d22","Type":"ContainerStarted","Data":"1f0fb6a0696bb35aba27a48e5b95aea97af5c07216c282ab7cd1925e39c6408c"} Oct 31 00:29:17.248562 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:17.248533 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pnmtp" event={"ID":"57decfb3-58bb-4b9c-b834-205594071bdf","Type":"ContainerStarted","Data":"290dfca05c359ae7d59ff7029b87f39951a55146fa46cfff9478e29183348f0e"} Oct 31 00:29:17.249720 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:17.249697 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xkqrm" event={"ID":"37dd17e5-ba62-4f95-bfc7-05b132104e44","Type":"ContainerStarted","Data":"eb1837caaa867184184cb737ed2ca7fbe06a6b484638b74d5fd579b1b5b896fd"} Oct 31 00:29:17.250919 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:17.250892 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f876fbf6-4mlmt" event={"ID":"91208431-95bb-4e24-ab4b-e0f80f0d6207","Type":"ContainerStarted","Data":"91572e8ab1e6067a5d2f4f9a1e81b1859c2d37d3048dda1a2a4aa345af7b7b2a"} Oct 31 00:29:18.262002 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:18.261849 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w49pc" event={"ID":"51dbc54b-218b-4274-9ebb-4cdbab400d22","Type":"ContainerStarted","Data":"ab7773d532fb45e2f56ac0c387ea475f52b7491080475c93cd7dd09e93cbb401"} Oct 31 00:29:19.266842 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:19.266560 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xkqrm" event={"ID":"37dd17e5-ba62-4f95-bfc7-05b132104e44","Type":"ContainerStarted","Data":"906faa1de7b3c105de7434341f317b2393b107efc309308ef9b2050071c313e7"} Oct 31 00:29:19.904507 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:19.904473 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2"] Oct 31 00:29:19.938700 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:19.938638 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2"] Oct 31 00:29:19.938700 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:19.938803 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:19.947559 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:19.947537 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40c70642-d50d-4f92-8ca1-10015c1a40cd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:19.947712 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:19.947644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40c70642-d50d-4f92-8ca1-10015c1a40cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:19.947712 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:19.947699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40c70642-d50d-4f92-8ca1-10015c1a40cd-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:19.947812 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:19.947774 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcx7m\" (UniqueName: \"kubernetes.io/projected/40c70642-d50d-4f92-8ca1-10015c1a40cd-kube-api-access-qcx7m\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:19.960592 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:19.960570 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:29:19.960862 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:19.960819 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:29:19.960964 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:19.960900 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:29:19.972335 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:19.972091 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:29:19.972335 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:19.972234 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wf9pg\"" Oct 31 00:29:19.972335 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:19.972310 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 00:29:19.977353 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:19.977331 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vl8l6"] Oct 31 00:29:19.996262 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:19.995775 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk"] Oct 31 00:29:19.996262 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:19.995920 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.023065 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.023036 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.027422 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.027400 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk"] Oct 31 00:29:20.050628 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.048421 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcx7m\" (UniqueName: \"kubernetes.io/projected/40c70642-d50d-4f92-8ca1-10015c1a40cd-kube-api-access-qcx7m\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:20.050628 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.048487 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40c70642-d50d-4f92-8ca1-10015c1a40cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:20.050628 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.048521 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d06594d-071d-47bb-ade5-16298fd91e2c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.050628 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.048550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-wtmp\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.050628 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.048578 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-tls\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.050628 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.048624 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bthrt\" (UniqueName: \"kubernetes.io/projected/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-api-access-bthrt\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.050628 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.048650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.050628 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.048697 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.050628 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.048732 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.050628 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.048769 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8d06594d-071d-47bb-ade5-16298fd91e2c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.050628 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.048796 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22g5c\" (UniqueName: \"kubernetes.io/projected/a4677743-13cb-4a3a-b1df-8bd6baef94f4-kube-api-access-22g5c\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.050628 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.048824 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-root\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.050628 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.048849 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-textfile\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.050628 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.048892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4677743-13cb-4a3a-b1df-8bd6baef94f4-metrics-client-ca\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.050628 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.048937 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-sys\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.051235 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.048966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40c70642-d50d-4f92-8ca1-10015c1a40cd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:20.051235 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.049015 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.051235 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.049049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40c70642-d50d-4f92-8ca1-10015c1a40cd-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:20.051235 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.049089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.051235 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.050580 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40c70642-d50d-4f92-8ca1-10015c1a40cd-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:20.054210 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.054174 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40c70642-d50d-4f92-8ca1-10015c1a40cd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:20.054644 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.054608 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40c70642-d50d-4f92-8ca1-10015c1a40cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:20.070003 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.069694 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:29:20.070003 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.069761 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lwv25\"" Oct 31 00:29:20.070003 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.069832 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 00:29:20.070003 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.069867 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:29:20.070003 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.069959 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kr8zh\"" Oct 31 00:29:20.070371 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.070218 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 00:29:20.070371 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.070287 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:29:20.070371 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.070346 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:29:20.077897 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.077875 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcx7m\" (UniqueName: \"kubernetes.io/projected/40c70642-d50d-4f92-8ca1-10015c1a40cd-kube-api-access-qcx7m\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:20.149556 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.149527 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.149661 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.149590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.150696 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.150035 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d06594d-071d-47bb-ade5-16298fd91e2c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.150696 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.150082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-wtmp\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.150696 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.150111 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-tls\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.150696 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.150154 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bthrt\" (UniqueName: \"kubernetes.io/projected/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-api-access-bthrt\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.150696 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.150179 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.150696 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.150214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.150696 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.150245 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.150696 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.150282 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8d06594d-071d-47bb-ade5-16298fd91e2c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.150696 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.150307 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22g5c\" (UniqueName: \"kubernetes.io/projected/a4677743-13cb-4a3a-b1df-8bd6baef94f4-kube-api-access-22g5c\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.150696 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.150336 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-root\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.150696 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.150358 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-textfile\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.150696 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.150386 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4677743-13cb-4a3a-b1df-8bd6baef94f4-metrics-client-ca\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.150696 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.150431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-sys\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.150696 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.150497 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-sys\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.150696 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.150584 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-wtmp\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.151362 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.150815 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d06594d-071d-47bb-ade5-16298fd91e2c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.151362 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.151073 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8d06594d-071d-47bb-ade5-16298fd91e2c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.151527 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.151480 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-root\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.151858 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.151837 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-textfile\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.152051 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.152009 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.152373 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.152349 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4677743-13cb-4a3a-b1df-8bd6baef94f4-metrics-client-ca\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.152509 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.152486 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.153295 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.153266 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.154135 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.154116 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.154625 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.154602 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.159012 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.158993 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-tls\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.168625 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.168600 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22g5c\" (UniqueName: \"kubernetes.io/projected/a4677743-13cb-4a3a-b1df-8bd6baef94f4-kube-api-access-22g5c\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.169099 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.169079 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bthrt\" (UniqueName: \"kubernetes.io/projected/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-api-access-bthrt\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.271930 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.271856 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w49pc" event={"ID":"51dbc54b-218b-4274-9ebb-4cdbab400d22","Type":"ContainerStarted","Data":"eae135c9ea6b9e0bf17a1dbbdf409eef9929f1f8d6f5248bcb47ffa956e32965"} Oct 31 00:29:20.275277 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.275249 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xkqrm" event={"ID":"37dd17e5-ba62-4f95-bfc7-05b132104e44","Type":"ContainerStarted","Data":"2c33aa8f4742d7f3df1f519b22cd06298ac0616a279759c9c5b9ff0e9821c3d2"} Oct 31 00:29:20.275484 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.275460 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:20.299935 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.299889 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w49pc" podStartSLOduration=1.9595774590000001 podStartE2EDuration="5.299874265s" totalImagesPullingTime="3.340296806s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:20.299202759 +0000 UTC m=+44.861162224" watchObservedRunningTime="2026-10-31 00:29:20.299874265 +0000 UTC m=+44.861833751" Oct 31 00:29:20.302542 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.302514 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:20.323371 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.323316 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-xkqrm" podStartSLOduration=2.770270465 podStartE2EDuration="5.32330302s" totalImagesPullingTime="2.553032555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:20.32268617 +0000 UTC m=+44.884645612" watchObservedRunningTime="2026-10-31 00:29:20.32330302 +0000 UTC m=+44.885262461" Oct 31 00:29:20.341837 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.341805 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:20.373788 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.370911 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:20.399450 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:29:20.399299 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4677743_13cb_4a3a_b1df_8bd6baef94f4.slice/crio-44dbf45ebf1a60401e63bee5572ce4fff1a74f91a4d83c6957d3940323784534 WatchSource:0}: Error finding container 44dbf45ebf1a60401e63bee5572ce4fff1a74f91a4d83c6957d3940323784534: Status 404 returned error can't find the container with id 44dbf45ebf1a60401e63bee5572ce4fff1a74f91a4d83c6957d3940323784534 Oct 31 00:29:20.600533 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.600404 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2"] Oct 31 00:29:20.614337 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:29:20.614316 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c70642_d50d_4f92_8ca1_10015c1a40cd.slice/crio-1389b5756470cd0ebf6e3e93cfb8814ab7e582ca290a67e548ceffba753a1933 WatchSource:0}: Error finding container 1389b5756470cd0ebf6e3e93cfb8814ab7e582ca290a67e548ceffba753a1933: Status 404 returned error can't find the container with id 1389b5756470cd0ebf6e3e93cfb8814ab7e582ca290a67e548ceffba753a1933 Oct 31 00:29:20.671941 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:20.671915 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk"] Oct 31 00:29:20.674996 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:29:20.674972 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d06594d_071d_47bb_ade5_16298fd91e2c.slice/crio-f40daf736160f81c37cdafce2c0fed75113a79f2955f39de17ecf4df7ed44f82 WatchSource:0}: Error finding container f40daf736160f81c37cdafce2c0fed75113a79f2955f39de17ecf4df7ed44f82: Status 404 returned error can't find the container with id f40daf736160f81c37cdafce2c0fed75113a79f2955f39de17ecf4df7ed44f82 Oct 31 00:29:21.079512 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.079264 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:29:21.093366 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.093339 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.101286 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.101258 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:29:21.101440 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.101390 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9gnrx\"" Oct 31 00:29:21.101967 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.101948 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:29:21.106010 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.105976 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:29:21.107153 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.107067 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:29:21.158966 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.158942 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.159104 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.158976 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc9ss\" (UniqueName: \"kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-kube-api-access-sc9ss\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.159104 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.159054 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-config-out\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.159104 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.159082 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-web-config\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.159199 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.159136 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.159199 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.159154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.159199 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.159170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.159199 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.159186 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.159336 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.159204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.159336 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.159298 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.159433 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.159377 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.159433 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.159421 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-config-volume\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.159495 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.159485 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.200158 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.200108 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:29:21.200158 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.200149 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:29:21.200358 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.200179 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:29:21.200358 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.200192 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:29:21.200358 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.200262 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:29:21.200358 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.200280 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:29:21.260174 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.260137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-config-volume\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.260352 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.260227 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.260430 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.260404 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.260489 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.260459 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc9ss\" (UniqueName: \"kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-kube-api-access-sc9ss\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.260540 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.260529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-config-out\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.260591 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.260569 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-web-config\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.260646 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.260612 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.260723 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.260642 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.260723 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.260691 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.260825 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.260726 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.260825 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.260753 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.260825 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.260784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.260825 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.260815 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.261646 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.261464 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.261646 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.261557 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.262387 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.262361 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.265517 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.265495 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-web-config\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.267202 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.267180 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.267202 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.267194 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-config-out\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.267503 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.267486 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.267692 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.267650 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.268137 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.268006 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-config-volume\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.268225 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.268065 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.268225 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.268081 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.268777 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.268755 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.275986 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.275966 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc9ss\" (UniqueName: \"kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-kube-api-access-sc9ss\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.288553 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.288524 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" event={"ID":"8d06594d-071d-47bb-ade5-16298fd91e2c","Type":"ContainerStarted","Data":"f40daf736160f81c37cdafce2c0fed75113a79f2955f39de17ecf4df7ed44f82"} Oct 31 00:29:21.289915 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.289888 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vl8l6" event={"ID":"a4677743-13cb-4a3a-b1df-8bd6baef94f4","Type":"ContainerStarted","Data":"44dbf45ebf1a60401e63bee5572ce4fff1a74f91a4d83c6957d3940323784534"} Oct 31 00:29:21.291962 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.291877 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" event={"ID":"40c70642-d50d-4f92-8ca1-10015c1a40cd","Type":"ContainerStarted","Data":"8bb8fcadbcb2f6e6f8b0549cda4831719e467650c1808f97e7e1855b8505eb82"} Oct 31 00:29:21.291962 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.291920 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" event={"ID":"40c70642-d50d-4f92-8ca1-10015c1a40cd","Type":"ContainerStarted","Data":"2d5bb1eeb686416e24ba786e2f92530731cdc8f9d70f68fd41339265253b1581"} Oct 31 00:29:21.291962 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.291939 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" event={"ID":"40c70642-d50d-4f92-8ca1-10015c1a40cd","Type":"ContainerStarted","Data":"1389b5756470cd0ebf6e3e93cfb8814ab7e582ca290a67e548ceffba753a1933"} Oct 31 00:29:21.293595 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.293572 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pnmtp" event={"ID":"57decfb3-58bb-4b9c-b834-205594071bdf","Type":"ContainerStarted","Data":"db82ab674dda757dfa0dd830a47347f118a8ca36753814dee23c49db0055a644"} Oct 31 00:29:21.295300 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.295278 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f876fbf6-4mlmt" event={"ID":"91208431-95bb-4e24-ab4b-e0f80f0d6207","Type":"ContainerStarted","Data":"01da56d3986382bf6399373b4f396d7863771f24c726394f1f379840f707417d"} Oct 31 00:29:21.336839 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.336720 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pnmtp" podStartSLOduration=2.282150785 podStartE2EDuration="6.336701446s" totalImagesPullingTime="4.054550661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:21.336147764 +0000 UTC m=+45.898107206" watchObservedRunningTime="2026-10-31 00:29:21.336701446 +0000 UTC m=+45.898660888" Oct 31 00:29:21.503722 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.503689 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:21.665158 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.665106 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-74f876fbf6-4mlmt" podStartSLOduration=2.733080483 podStartE2EDuration="6.665093278s" totalImagesPullingTime="3.932012795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:21.387971511 +0000 UTC m=+45.949930955" watchObservedRunningTime="2026-10-31 00:29:21.665093278 +0000 UTC m=+46.227052719" Oct 31 00:29:21.666161 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.666138 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:29:21.790208 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:29:21.790170 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa459c8_39bb_4020_a514_0e03536a3dda.slice/crio-acaf343745ccfd10174cdb916b6488c6fe4fb0428d40402e6c863d3e03a9f596 WatchSource:0}: Error finding container acaf343745ccfd10174cdb916b6488c6fe4fb0428d40402e6c863d3e03a9f596: Status 404 returned error can't find the container with id acaf343745ccfd10174cdb916b6488c6fe4fb0428d40402e6c863d3e03a9f596 Oct 31 00:29:21.962465 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.961975 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7bcdf957ff-s472c"] Oct 31 00:29:21.965511 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.965202 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:21.978184 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.977896 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 00:29:21.978184 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.977915 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 00:29:21.978184 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.977997 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 00:29:21.978184 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.978050 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 00:29:21.978184 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.978094 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-n2kx5\"" Oct 31 00:29:21.978184 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.978142 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 00:29:21.978184 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.978154 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1pefrp1lki40c\"" Oct 31 00:29:21.989764 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:21.989741 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bcdf957ff-s472c"] Oct 31 00:29:22.070000 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.069967 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-grpc-tls\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.070173 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.070023 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.070173 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.070052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmzx8\" (UniqueName: \"kubernetes.io/projected/86e13c77-adcc-4f59-b1a4-93d090e2379e-kube-api-access-bmzx8\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.070173 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.070138 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86e13c77-adcc-4f59-b1a4-93d090e2379e-tmp\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.070307 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.070191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.070307 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.070217 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-tls\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.070307 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.070248 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.070307 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.070303 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.070501 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.070330 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86e13c77-adcc-4f59-b1a4-93d090e2379e-metrics-client-ca\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.171582 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.171550 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.171784 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.171610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86e13c77-adcc-4f59-b1a4-93d090e2379e-metrics-client-ca\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.171784 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.171637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-grpc-tls\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.171784 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.171683 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.171784 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.171708 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmzx8\" (UniqueName: \"kubernetes.io/projected/86e13c77-adcc-4f59-b1a4-93d090e2379e-kube-api-access-bmzx8\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.171784 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.171729 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86e13c77-adcc-4f59-b1a4-93d090e2379e-tmp\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.171784 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.171761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.171784 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.171784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-tls\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.172140 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.171804 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.172279 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.172255 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86e13c77-adcc-4f59-b1a4-93d090e2379e-tmp\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.172595 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.172559 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86e13c77-adcc-4f59-b1a4-93d090e2379e-metrics-client-ca\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.175941 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.175910 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.176064 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.175919 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-grpc-tls\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.176322 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.176302 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.177136 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.177106 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.177233 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.177110 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.177291 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.177241 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-tls\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.187048 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.187030 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmzx8\" (UniqueName: \"kubernetes.io/projected/86e13c77-adcc-4f59-b1a4-93d090e2379e-kube-api-access-bmzx8\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.286966 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.286933 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:22.302924 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.302898 2575 generic.go:356] "Generic (PLEG): container finished" podID="a4677743-13cb-4a3a-b1df-8bd6baef94f4" containerID="135b24dfdec8cceba8ebbed7ecf5d5bca0766df94513156ae96bce986989b1b9" exitCode=0 Oct 31 00:29:22.303048 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.302960 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vl8l6" event={"ID":"a4677743-13cb-4a3a-b1df-8bd6baef94f4","Type":"ContainerDied","Data":"135b24dfdec8cceba8ebbed7ecf5d5bca0766df94513156ae96bce986989b1b9"} Oct 31 00:29:22.305467 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.305406 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" event={"ID":"40c70642-d50d-4f92-8ca1-10015c1a40cd","Type":"ContainerStarted","Data":"fd2495650f37720b56cd945bd73cc3c01d9c34a38194074dda770e8187ceb03f"} Oct 31 00:29:22.306756 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.306724 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bfa459c8-39bb-4020-a514-0e03536a3dda","Type":"ContainerStarted","Data":"acaf343745ccfd10174cdb916b6488c6fe4fb0428d40402e6c863d3e03a9f596"} Oct 31 00:29:22.585157 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.584870 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" podStartSLOduration=2.184420923 podStartE2EDuration="3.584847894s" totalImagesPullingTime="1.400426971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:22.358892713 +0000 UTC m=+46.920852166" watchObservedRunningTime="2026-10-31 00:29:22.584847894 +0000 UTC m=+47.146807337" Oct 31 00:29:22.586084 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:22.586056 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bcdf957ff-s472c"] Oct 31 00:29:22.589404 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:29:22.589379 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e13c77_adcc_4f59_b1a4_93d090e2379e.slice/crio-d99f9a3644d63d499fbc5ebf8b9687ca9faec6c7acdc28dbf6758af61e812b72 WatchSource:0}: Error finding container d99f9a3644d63d499fbc5ebf8b9687ca9faec6c7acdc28dbf6758af61e812b72: Status 404 returned error can't find the container with id d99f9a3644d63d499fbc5ebf8b9687ca9faec6c7acdc28dbf6758af61e812b72 Oct 31 00:29:23.312049 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:23.311956 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" event={"ID":"86e13c77-adcc-4f59-b1a4-93d090e2379e","Type":"ContainerStarted","Data":"d99f9a3644d63d499fbc5ebf8b9687ca9faec6c7acdc28dbf6758af61e812b72"} Oct 31 00:29:23.314440 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:23.314409 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" event={"ID":"8d06594d-071d-47bb-ade5-16298fd91e2c","Type":"ContainerStarted","Data":"14e76783b12ffdab8c3d3bc03d260dbbbb2f965dc73f669be7f2125d6cff7090"} Oct 31 00:29:23.314573 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:23.314461 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" event={"ID":"8d06594d-071d-47bb-ade5-16298fd91e2c","Type":"ContainerStarted","Data":"ed25eb968025a413f75e6ca3ecd91bec33918c85a3340e7682f6ed0baa33baf3"} Oct 31 00:29:23.314573 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:23.314490 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" event={"ID":"8d06594d-071d-47bb-ade5-16298fd91e2c","Type":"ContainerStarted","Data":"ea07f1b459c28fc44a14c618530b80f78885f10158f61fc6a6f0dd9b2bae520a"} Oct 31 00:29:23.316973 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:23.316947 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vl8l6" event={"ID":"a4677743-13cb-4a3a-b1df-8bd6baef94f4","Type":"ContainerStarted","Data":"520f24bb5ade0584d40ad9bbab95bd405f11c45caf8d8b37e86f56f3b3f7ff58"} Oct 31 00:29:23.317092 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:23.316990 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vl8l6" event={"ID":"a4677743-13cb-4a3a-b1df-8bd6baef94f4","Type":"ContainerStarted","Data":"fe62ffccf0a88d7f96c2ef42fc5c7de1c8c387985ca699e7fabb3a8bdadf7169"} Oct 31 00:29:23.318658 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:23.318633 2575 generic.go:356] "Generic (PLEG): container finished" podID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerID="fb92b869942ab49a2d6edb574af10bc1fcb0d928852268c7469de8726e518eef" exitCode=0 Oct 31 00:29:23.318777 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:23.318700 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bfa459c8-39bb-4020-a514-0e03536a3dda","Type":"ContainerDied","Data":"fb92b869942ab49a2d6edb574af10bc1fcb0d928852268c7469de8726e518eef"} Oct 31 00:29:23.341585 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:23.341534 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" podStartSLOduration=2.757375946 podStartE2EDuration="4.3415158s" totalImagesPullingTime="1.584139854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:23.341035624 +0000 UTC m=+47.902995067" watchObservedRunningTime="2026-10-31 00:29:23.3415158 +0000 UTC m=+47.903475244" Oct 31 00:29:23.372859 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:23.372807 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vl8l6" podStartSLOduration=2.862458942 podStartE2EDuration="4.372792579s" totalImagesPullingTime="1.419172438s" totalInitContainerRuntime="91.161199ms" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:23.372624682 +0000 UTC m=+47.934584128" watchObservedRunningTime="2026-10-31 00:29:23.372792579 +0000 UTC m=+47.934752015" Oct 31 00:29:24.390253 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.390222 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-67cc7bf54c-zxwf4"] Oct 31 00:29:24.392508 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.392477 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.400018 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.399994 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 00:29:24.401177 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.401154 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 00:29:24.401396 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.401379 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 00:29:24.401718 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.401639 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2imsi3d4ipqml\"" Oct 31 00:29:24.401776 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.401746 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bzqj7\"" Oct 31 00:29:24.401817 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.401791 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 00:29:24.408758 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.408660 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67cc7bf54c-zxwf4"] Oct 31 00:29:24.493642 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.493604 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b2eb2a42-2573-414d-8516-aac46f430550-audit-log\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.493826 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.493699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-secret-metrics-server-tls\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.493826 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.493730 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2eb2a42-2573-414d-8516-aac46f430550-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.493826 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.493762 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b2eb2a42-2573-414d-8516-aac46f430550-metrics-server-audit-profiles\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.493826 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.493798 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-secret-metrics-server-client-certs\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.493826 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.493813 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwjlt\" (UniqueName: \"kubernetes.io/projected/b2eb2a42-2573-414d-8516-aac46f430550-kube-api-access-xwjlt\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.494050 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.493841 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-client-ca-bundle\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.594295 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.594266 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-secret-metrics-server-client-certs\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.594389 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.594312 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwjlt\" (UniqueName: \"kubernetes.io/projected/b2eb2a42-2573-414d-8516-aac46f430550-kube-api-access-xwjlt\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.594389 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.594356 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-client-ca-bundle\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.594513 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.594411 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b2eb2a42-2573-414d-8516-aac46f430550-audit-log\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.594513 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.594481 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-secret-metrics-server-tls\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.594618 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.594514 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2eb2a42-2573-414d-8516-aac46f430550-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.594618 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.594559 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b2eb2a42-2573-414d-8516-aac46f430550-metrics-server-audit-profiles\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.595977 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.595862 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2eb2a42-2573-414d-8516-aac46f430550-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.595977 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.595883 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b2eb2a42-2573-414d-8516-aac46f430550-audit-log\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.596208 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.596174 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b2eb2a42-2573-414d-8516-aac46f430550-metrics-server-audit-profiles\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.603475 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.603364 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-secret-metrics-server-client-certs\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.604149 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.604128 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-secret-metrics-server-tls\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.608397 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.608373 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-client-ca-bundle\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.612247 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.612228 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwjlt\" (UniqueName: \"kubernetes.io/projected/b2eb2a42-2573-414d-8516-aac46f430550-kube-api-access-xwjlt\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.678241 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.678214 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-65wl9"] Oct 31 00:29:24.681597 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.680983 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" Oct 31 00:29:24.687778 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.687760 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 00:29:24.687872 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.687846 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pdw62\"" Oct 31 00:29:24.696147 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.696124 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/266a13d4-3f45-44d6-ac0e-3e171ce4e928-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-65wl9\" (UID: \"266a13d4-3f45-44d6-ac0e-3e171ce4e928\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" Oct 31 00:29:24.696254 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.696170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29dqz\" (UniqueName: \"kubernetes.io/projected/266a13d4-3f45-44d6-ac0e-3e171ce4e928-kube-api-access-29dqz\") pod \"monitoring-plugin-77f4f76594-65wl9\" (UID: \"266a13d4-3f45-44d6-ac0e-3e171ce4e928\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" Oct 31 00:29:24.699082 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.699060 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-65wl9"] Oct 31 00:29:24.707507 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.707487 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:24.797814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.797472 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29dqz\" (UniqueName: \"kubernetes.io/projected/266a13d4-3f45-44d6-ac0e-3e171ce4e928-kube-api-access-29dqz\") pod \"monitoring-plugin-77f4f76594-65wl9\" (UID: \"266a13d4-3f45-44d6-ac0e-3e171ce4e928\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" Oct 31 00:29:24.797814 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.797576 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/266a13d4-3f45-44d6-ac0e-3e171ce4e928-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-65wl9\" (UID: \"266a13d4-3f45-44d6-ac0e-3e171ce4e928\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" Oct 31 00:29:24.800216 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.800190 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/266a13d4-3f45-44d6-ac0e-3e171ce4e928-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-65wl9\" (UID: \"266a13d4-3f45-44d6-ac0e-3e171ce4e928\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" Oct 31 00:29:24.811966 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.811946 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29dqz\" (UniqueName: \"kubernetes.io/projected/266a13d4-3f45-44d6-ac0e-3e171ce4e928-kube-api-access-29dqz\") pod \"monitoring-plugin-77f4f76594-65wl9\" (UID: \"266a13d4-3f45-44d6-ac0e-3e171ce4e928\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" Oct 31 00:29:24.993155 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:24.993130 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67cc7bf54c-zxwf4"] Oct 31 00:29:24.996819 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:29:24.996793 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2eb2a42_2573_414d_8516_aac46f430550.slice/crio-3de12aaf77de82f2be7323f69e46b04a88db46512e6a234722126ffd81fb1775 WatchSource:0}: Error finding container 3de12aaf77de82f2be7323f69e46b04a88db46512e6a234722126ffd81fb1775: Status 404 returned error can't find the container with id 3de12aaf77de82f2be7323f69e46b04a88db46512e6a234722126ffd81fb1775 Oct 31 00:29:25.008529 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:25.008512 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" Oct 31 00:29:25.294590 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:25.294519 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-65wl9"] Oct 31 00:29:25.297196 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:29:25.297172 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266a13d4_3f45_44d6_ac0e_3e171ce4e928.slice/crio-80ffc814accc550d820b6b4f4723f239b1bf007502c43055fae6a38b2b7b1903 WatchSource:0}: Error finding container 80ffc814accc550d820b6b4f4723f239b1bf007502c43055fae6a38b2b7b1903: Status 404 returned error can't find the container with id 80ffc814accc550d820b6b4f4723f239b1bf007502c43055fae6a38b2b7b1903 Oct 31 00:29:25.327881 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:25.327852 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" event={"ID":"86e13c77-adcc-4f59-b1a4-93d090e2379e","Type":"ContainerStarted","Data":"be32e38e31e541b79aeb6c44f7583a812ee4aad9708572dd65accae66240a9b4"} Oct 31 00:29:25.327993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:25.327931 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" event={"ID":"86e13c77-adcc-4f59-b1a4-93d090e2379e","Type":"ContainerStarted","Data":"993b519bb97c95654c0d346600c09464cbb1ca9303e1d4d9181b3cbbb6894e66"} Oct 31 00:29:25.327993 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:25.327975 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" event={"ID":"86e13c77-adcc-4f59-b1a4-93d090e2379e","Type":"ContainerStarted","Data":"36855113ec3715e316d02c44c1fcae66c3e7fc8dfaf015a10a08a32b8ddf2356"} Oct 31 00:29:25.330990 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:25.330969 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bfa459c8-39bb-4020-a514-0e03536a3dda","Type":"ContainerStarted","Data":"f65e4e8455ee99e23dd7c759b18664c4a505a28fc815a22e8fd101dd01ea654f"} Oct 31 00:29:25.331106 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:25.331031 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bfa459c8-39bb-4020-a514-0e03536a3dda","Type":"ContainerStarted","Data":"d1259cbc9034b86982202e36778e9c3115b54e49f6a26d09d1b5ff34b05faa8d"} Oct 31 00:29:25.331106 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:25.331073 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bfa459c8-39bb-4020-a514-0e03536a3dda","Type":"ContainerStarted","Data":"c987cd437379dc739d69eab5766ddbed2c4a612117986fe268fdd2056899ce9c"} Oct 31 00:29:25.331172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:25.331112 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bfa459c8-39bb-4020-a514-0e03536a3dda","Type":"ContainerStarted","Data":"1e800c60b24a1ab552d955413f9f8b2ab2a3540b3d28f27face081d026cddc03"} Oct 31 00:29:25.331172 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:25.331149 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bfa459c8-39bb-4020-a514-0e03536a3dda","Type":"ContainerStarted","Data":"0f52f91ae90659018cb48c193137efa09e1b312f780516f36c5c6d53522f8cf7"} Oct 31 00:29:25.332253 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:25.332234 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" event={"ID":"266a13d4-3f45-44d6-ac0e-3e171ce4e928","Type":"ContainerStarted","Data":"80ffc814accc550d820b6b4f4723f239b1bf007502c43055fae6a38b2b7b1903"} Oct 31 00:29:25.333333 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:25.333317 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" event={"ID":"b2eb2a42-2573-414d-8516-aac46f430550","Type":"ContainerStarted","Data":"3de12aaf77de82f2be7323f69e46b04a88db46512e6a234722126ffd81fb1775"} Oct 31 00:29:26.319613 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.319577 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:29:26.324184 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.324161 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.332837 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.332817 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:29:26.332997 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.332984 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:29:26.333196 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.333172 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:29:26.333297 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.333203 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:29:26.333297 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.333283 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:29:26.333377 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.333284 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:29:26.333453 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.333436 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:29:26.333501 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.333488 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:29:26.333549 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.333521 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:29:26.333591 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.333571 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:29:26.333642 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.333614 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:29:26.334863 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.334845 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:29:26.334969 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.334950 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5q9q9rm8kcsp5\"" Oct 31 00:29:26.335738 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.335724 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-84jxj\"" Oct 31 00:29:26.335875 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.335861 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:29:26.336894 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.336877 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:29:26.346254 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.346229 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:29:26.357507 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.357426 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bfa459c8-39bb-4020-a514-0e03536a3dda","Type":"ContainerStarted","Data":"cf709572a498967a1d01ac5976ebdc5c977f8ae769a867cfe17ffdde910cb80a"} Oct 31 00:29:26.362133 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.362106 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" event={"ID":"86e13c77-adcc-4f59-b1a4-93d090e2379e","Type":"ContainerStarted","Data":"344e731f3c3dc95d6c38604f807cd4a105bcafaec4cc3ddf35509b1cd31ed8d7"} Oct 31 00:29:26.362251 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.362172 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" event={"ID":"86e13c77-adcc-4f59-b1a4-93d090e2379e","Type":"ContainerStarted","Data":"ff5820f557698dcff870d1f0888e8ff86ef45971d4ec7e8144779e9cfaf2737d"} Oct 31 00:29:26.362251 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.362205 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" event={"ID":"86e13c77-adcc-4f59-b1a4-93d090e2379e","Type":"ContainerStarted","Data":"76b2cb94318eadf2fc8e482c667dd656114d37447612c92147f6ab487f009c06"} Oct 31 00:29:26.362411 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.362342 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:26.398587 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.398554 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:26.398828 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.398614 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:26.404830 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.404803 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:26.408246 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.408148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.408246 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.408212 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.408428 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.408406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.408524 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.408505 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.408578 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.408558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbxs7\" (UniqueName: \"kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-kube-api-access-tbxs7\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.408703 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.408656 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-config\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.408780 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.408755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.408838 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.408792 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.408922 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.408903 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.409014 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.408992 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.409066 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.409030 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.409113 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.409079 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.409169 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.409150 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.409233 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.409216 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.409297 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.409280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.409345 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.409328 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.409396 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.409355 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-web-config\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.409430 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.409396 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.409475 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.409460 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.409523 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.409509 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-config-out\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.422428 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.422376 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" podStartSLOduration=2.179399389 podStartE2EDuration="5.422358407s" totalImagesPullingTime="3.242959018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:26.421219805 +0000 UTC m=+50.983179247" watchObservedRunningTime="2026-10-31 00:29:26.422358407 +0000 UTC m=+50.984317850" Oct 31 00:29:26.460990 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.460941 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.928478411 podStartE2EDuration="5.460927461s" totalImagesPullingTime="3.519839635s" totalInitContainerRuntime="12.609415ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:26.460232848 +0000 UTC m=+51.022192290" watchObservedRunningTime="2026-10-31 00:29:26.460927461 +0000 UTC m=+51.022886902" Oct 31 00:29:26.511659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.510721 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.511659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.510754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.511659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.510809 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.511659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.510862 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.511659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.510889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.511659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.510918 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.511659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.510980 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.511659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.511028 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.511659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.511081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.511659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.511111 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.511659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.511135 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-web-config\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.511659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.511163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.511659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.511197 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.511659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.511232 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-config-out\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.511659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.511259 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.511659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.511287 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.511659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.511325 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.512661 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.511374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.512661 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.511423 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbxs7\" (UniqueName: \"kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-kube-api-access-tbxs7\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.512661 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.511470 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-config\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.512661 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.511545 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.512661 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.512107 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.512661 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.512460 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.513868 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.513137 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.513868 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.513727 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.514030 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.513931 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.514355 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.514319 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.515532 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.514953 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.515532 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.515390 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.516420 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.516392 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.517621 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.517576 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-config-out\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.518060 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.518030 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.518496 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.518466 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-config\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.518716 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.518696 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.518853 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.518838 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.519104 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.519088 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.519399 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.519382 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.519675 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.519643 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-web-config\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.522290 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.522269 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.542096 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.542070 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbxs7\" (UniqueName: \"kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-kube-api-access-tbxs7\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.657651 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.657619 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:26.962820 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:26.957630 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:29:26.968835 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:29:26.968810 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc16559_1d24_40ea_9ba6_aa6896178faf.slice/crio-c45165966107cab5eea0f3a82ed97e285ea82de8829878df0a678cee5634d91c WatchSource:0}: Error finding container c45165966107cab5eea0f3a82ed97e285ea82de8829878df0a678cee5634d91c: Status 404 returned error can't find the container with id c45165966107cab5eea0f3a82ed97e285ea82de8829878df0a678cee5634d91c Oct 31 00:29:27.369430 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:27.369395 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" event={"ID":"266a13d4-3f45-44d6-ac0e-3e171ce4e928","Type":"ContainerStarted","Data":"fecd26efb1992f8636180a17d1b433c9a1f9c094d09bb726b2fa778a76c03164"} Oct 31 00:29:27.369871 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:27.369567 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" Oct 31 00:29:27.371112 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:27.371085 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" event={"ID":"b2eb2a42-2573-414d-8516-aac46f430550","Type":"ContainerStarted","Data":"89f507fa40e83a9633928631e32880048396daa0a3cf542cb3d57e072edc94cb"} Oct 31 00:29:27.372762 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:27.372741 2575 generic.go:356] "Generic (PLEG): container finished" podID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerID="eb74ac6c958294b3fdf7f5e08d3c3921e026e9c566c85820664a00f8705ccb49" exitCode=0 Oct 31 00:29:27.372881 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:27.372845 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffc16559-1d24-40ea-9ba6-aa6896178faf","Type":"ContainerDied","Data":"eb74ac6c958294b3fdf7f5e08d3c3921e026e9c566c85820664a00f8705ccb49"} Oct 31 00:29:27.372950 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:27.372903 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffc16559-1d24-40ea-9ba6-aa6896178faf","Type":"ContainerStarted","Data":"c45165966107cab5eea0f3a82ed97e285ea82de8829878df0a678cee5634d91c"} Oct 31 00:29:27.375802 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:27.375785 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" Oct 31 00:29:27.377485 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:27.377469 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:27.392106 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:27.392071 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" podStartSLOduration=1.750573722 podStartE2EDuration="3.392060951s" totalImagesPullingTime="1.641487229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:27.391056382 +0000 UTC m=+51.953015823" watchObservedRunningTime="2026-10-31 00:29:27.392060951 +0000 UTC m=+51.954020395" Oct 31 00:29:27.504894 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:27.504845 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" podStartSLOduration=1.80671128 podStartE2EDuration="3.504832433s" totalImagesPullingTime="1.698121153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:27.503823204 +0000 UTC m=+52.065782645" watchObservedRunningTime="2026-10-31 00:29:27.504832433 +0000 UTC m=+52.066791905" Oct 31 00:29:29.218936 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:29.218908 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:30.298105 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:30.298076 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:31.390981 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:31.390947 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffc16559-1d24-40ea-9ba6-aa6896178faf","Type":"ContainerStarted","Data":"e5db3ac7f06883e8365d3b6d52e6ee8ac52f98637c43bbe56640fe0b32fe6ad8"} Oct 31 00:29:31.391317 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:31.391027 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffc16559-1d24-40ea-9ba6-aa6896178faf","Type":"ContainerStarted","Data":"30600e3dbe12f6b65c49307713f4f1628da379e8962f6faa812f3993e402e2c8"} Oct 31 00:29:31.391317 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:31.391068 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffc16559-1d24-40ea-9ba6-aa6896178faf","Type":"ContainerStarted","Data":"5f183a8c26cf96890f3cf614f2f1425529ff8f38aeb87c475440d1277fde8e16"} Oct 31 00:29:31.391317 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:31.391111 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffc16559-1d24-40ea-9ba6-aa6896178faf","Type":"ContainerStarted","Data":"81cac64659d6aa22472fe0bbbf50f05d1b9d5b9b2b1290a60883f4fe9401aa6b"} Oct 31 00:29:31.391317 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:31.391147 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffc16559-1d24-40ea-9ba6-aa6896178faf","Type":"ContainerStarted","Data":"93356c90d027466eb63fd46907afa38a144ab942bcf1276f1fedb82aba60c243"} Oct 31 00:29:31.391317 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:31.391181 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffc16559-1d24-40ea-9ba6-aa6896178faf","Type":"ContainerStarted","Data":"f9af75c3923d6d5d6b484625c5f9200ac1099246e2d32f96fbbfbe767f49b930"} Oct 31 00:29:31.439810 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:31.439747 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.455156614 podStartE2EDuration="5.439724729s" totalImagesPullingTime="2.962436013s" totalInitContainerRuntime="22.132102ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:31.438661651 +0000 UTC m=+56.000621096" watchObservedRunningTime="2026-10-31 00:29:31.439724729 +0000 UTC m=+56.001684173" Oct 31 00:29:31.658386 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:31.658312 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:32.379971 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:32.379944 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:41.646950 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:41.646919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs\") pod \"network-metrics-daemon-jcdhx\" (UID: \"ba178af9-0f61-4c7b-a5ab-36e772495016\") " pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:29:41.680785 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:41.680761 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:29:41.689961 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:41.689935 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs\") pod \"network-metrics-daemon-jcdhx\" (UID: \"ba178af9-0f61-4c7b-a5ab-36e772495016\") " pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:29:41.747407 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:41.747382 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:41.750966 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:41.750950 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:29:41.759989 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:41.759962 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:41.848624 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:41.848598 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lkc\" (UniqueName: \"kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc\") pod \"network-check-target-zwhbr\" (UID: \"6f86ab44-a957-4e07-8bdf-5a8f813f27d4\") " pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:41.849659 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:41.849642 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4bqfb\"" Oct 31 00:29:41.852050 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:41.852028 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:29:41.857122 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:41.857108 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:41.862374 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:41.862361 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:29:41.871655 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:41.871634 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7lkc\" (UniqueName: \"kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc\") pod \"network-check-target-zwhbr\" (UID: \"6f86ab44-a957-4e07-8bdf-5a8f813f27d4\") " pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:41.886552 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:41.886533 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hxhg5\"" Oct 31 00:29:41.893597 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:41.893569 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:29:41.948106 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:41.948084 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qx27s\"" Oct 31 00:29:41.956044 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:41.955828 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:42.153057 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:42.153026 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dvl2d"] Oct 31 00:29:42.155625 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:29:42.155597 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b81351_0c71_49b6_bd86_e48483b8b555.slice/crio-b69fd75ea1b90792846d487eba2508eb6db2af1ee90053514322be191f96b622 WatchSource:0}: Error finding container b69fd75ea1b90792846d487eba2508eb6db2af1ee90053514322be191f96b622: Status 404 returned error can't find the container with id b69fd75ea1b90792846d487eba2508eb6db2af1ee90053514322be191f96b622 Oct 31 00:29:42.187512 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:42.187488 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jcdhx"] Oct 31 00:29:42.191199 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:29:42.191171 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba178af9_0f61_4c7b_a5ab_36e772495016.slice/crio-371eb6cd0849d585a4ab3d88b1ea5b40529fe1f3d8cbb61047ae753cd5844dba WatchSource:0}: Error finding container 371eb6cd0849d585a4ab3d88b1ea5b40529fe1f3d8cbb61047ae753cd5844dba: Status 404 returned error can't find the container with id 371eb6cd0849d585a4ab3d88b1ea5b40529fe1f3d8cbb61047ae753cd5844dba Oct 31 00:29:42.237968 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:42.237936 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zwhbr"] Oct 31 00:29:42.240677 ip-10-0-143-213 kubenswrapper[2575]: W1031 00:29:42.240640 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f86ab44_a957_4e07_8bdf_5a8f813f27d4.slice/crio-c5ea6728d5c08fe5bc5e5c62b3219740867b0e727c7faa521ad1ac74cfda5146 WatchSource:0}: Error finding container c5ea6728d5c08fe5bc5e5c62b3219740867b0e727c7faa521ad1ac74cfda5146: Status 404 returned error can't find the container with id c5ea6728d5c08fe5bc5e5c62b3219740867b0e727c7faa521ad1ac74cfda5146 Oct 31 00:29:42.434864 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:42.434784 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zwhbr" event={"ID":"6f86ab44-a957-4e07-8bdf-5a8f813f27d4","Type":"ContainerStarted","Data":"c5ea6728d5c08fe5bc5e5c62b3219740867b0e727c7faa521ad1ac74cfda5146"} Oct 31 00:29:42.436006 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:42.435982 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jcdhx" event={"ID":"ba178af9-0f61-4c7b-a5ab-36e772495016","Type":"ContainerStarted","Data":"371eb6cd0849d585a4ab3d88b1ea5b40529fe1f3d8cbb61047ae753cd5844dba"} Oct 31 00:29:42.437149 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:42.437128 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dvl2d" event={"ID":"e7b81351-0c71-49b6-bd86-e48483b8b555","Type":"ContainerStarted","Data":"b69fd75ea1b90792846d487eba2508eb6db2af1ee90053514322be191f96b622"} Oct 31 00:29:43.445014 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:43.444981 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jcdhx" event={"ID":"ba178af9-0f61-4c7b-a5ab-36e772495016","Type":"ContainerStarted","Data":"0c82b58efc44d32d13c6c0a30ddcdea4ec733c05918e1cb14f3e09ae9cd7dae6"} Oct 31 00:29:44.450911 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:44.450872 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jcdhx" event={"ID":"ba178af9-0f61-4c7b-a5ab-36e772495016","Type":"ContainerStarted","Data":"d767dc12235feedccb0ec2bc741994f6c0a914ed9ac116e97d3af268239651d2"} Oct 31 00:29:44.480953 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:44.480890 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jcdhx" podStartSLOduration=68.357506317 podStartE2EDuration="1m9.480870564s" totalImagesPullingTime="1.123364247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:44.480478991 +0000 UTC m=+69.042438437" watchObservedRunningTime="2026-10-31 00:29:44.480870564 +0000 UTC m=+69.042830008" Oct 31 00:29:44.708063 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:44.707977 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:44.708063 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:44.708047 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:46.302159 ip-10-0-143-213 kubenswrapper[2575]: E1031 00:29:46.302119 2575 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:29:46.461047 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:46.461001 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zwhbr" event={"ID":"6f86ab44-a957-4e07-8bdf-5a8f813f27d4","Type":"ContainerStarted","Data":"5f8dc8d8bc929c96176285fca2b00e2f754c2c441c66661df331a7e645b8c310"} Oct 31 00:29:46.461287 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:46.461266 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:46.548642 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:46.548583 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zwhbr" podStartSLOduration=67.157457944 podStartE2EDuration="1m10.548564442s" totalImagesPullingTime="3.391106498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:46.484083582 +0000 UTC m=+71.046043023" watchObservedRunningTime="2026-10-31 00:29:46.548564442 +0000 UTC m=+71.110523886" Oct 31 00:29:46.550278 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:46.550253 2575 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74f876fbf6-4mlmt"] Oct 31 00:29:47.457056 ip-10-0-143-213 kubenswrapper[2575]: I1031 00:29:47.456815 2575 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:29:47.456963 ip-10-0-143-213 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 00:29:47.464228 ip-10-0-143-213 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 00:29:47.464464 ip-10-0-143-213 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 00:29:47.464662 ip-10-0-143-213 systemd[1]: kubelet.service: Consumed 4.520s CPU time. Oct 31 00:29:47.477176 ip-10-0-143-213 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:29:47.544436 ip-10-0-143-213 kubenswrapper[5939]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:29:47.544436 ip-10-0-143-213 kubenswrapper[5939]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:29:47.544436 ip-10-0-143-213 kubenswrapper[5939]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:29:47.544436 ip-10-0-143-213 kubenswrapper[5939]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:29:47.546488 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546408 5939 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:29:47.546488 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546488 5939 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546493 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546497 5939 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546500 5939 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546504 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546507 5939 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546510 5939 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546513 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546515 5939 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546518 5939 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546521 5939 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546523 5939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546526 5939 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546529 5939 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546531 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546534 5939 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546536 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546539 5939 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546542 5939 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546545 5939 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:29:47.546554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546548 5939 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546550 5939 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546553 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546556 5939 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546559 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546564 5939 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546568 5939 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546571 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546574 5939 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546577 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546579 5939 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546582 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546585 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546588 5939 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546591 5939 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546593 5939 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546596 5939 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546599 5939 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546601 5939 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:29:47.547057 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546604 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546607 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546609 5939 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546612 5939 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546614 5939 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546617 5939 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546619 5939 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546622 5939 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546625 5939 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546628 5939 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546631 5939 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546633 5939 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546636 5939 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546639 5939 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546641 5939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546645 5939 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546647 5939 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546650 5939 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546652 5939 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546655 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:29:47.547526 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546658 5939 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546661 5939 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546679 5939 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546683 5939 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546687 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546691 5939 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546694 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546696 5939 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546699 5939 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546701 5939 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546704 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546707 5939 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546710 5939 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546713 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546715 5939 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546718 5939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546721 5939 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546723 5939 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546726 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546728 5939 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:29:47.548021 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546731 5939 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546734 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546737 5939 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546739 5939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546742 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546744 5939 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546748 5939 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546753 5939 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546756 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546759 5939 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546761 5939 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546764 5939 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546767 5939 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546769 5939 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546772 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546774 5939 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546778 5939 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546781 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546783 5939 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:29:47.548539 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546786 5939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546788 5939 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546791 5939 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546793 5939 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546796 5939 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546799 5939 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.546801 5939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547561 5939 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547568 5939 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547571 5939 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547575 5939 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547578 5939 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547581 5939 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547584 5939 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547587 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547589 5939 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547592 5939 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547594 5939 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547597 5939 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:29:47.549034 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547600 5939 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547603 5939 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547606 5939 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547608 5939 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547611 5939 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547613 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547615 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547618 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547621 5939 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547623 5939 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547627 5939 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547631 5939 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547634 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547637 5939 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547640 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547644 5939 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547646 5939 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547649 5939 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547652 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:29:47.549600 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547655 5939 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547658 5939 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547660 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547680 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547683 5939 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547685 5939 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547688 5939 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547691 5939 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547693 5939 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547696 5939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547699 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547701 5939 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547704 5939 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547706 5939 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547710 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547712 5939 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547715 5939 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547717 5939 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547720 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547722 5939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:29:47.550084 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547725 5939 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547727 5939 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547730 5939 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547733 5939 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547735 5939 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547738 5939 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547740 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547743 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547746 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547748 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547751 5939 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547753 5939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547756 5939 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547759 5939 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547762 5939 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547764 5939 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547767 5939 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547769 5939 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547772 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547775 5939 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:29:47.550579 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547779 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547781 5939 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547784 5939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547786 5939 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547789 5939 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547791 5939 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547795 5939 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547798 5939 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547800 5939 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547803 5939 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547806 5939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547808 5939 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547811 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547813 5939 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547816 5939 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547818 5939 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547821 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547823 5939 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547826 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547828 5939 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:29:47.551433 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547831 5939 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547834 5939 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547836 5939 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547839 5939 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547842 5939 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547844 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547847 5939 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547850 5939 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547853 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547858 5939 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547862 5939 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547865 5939 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547868 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547870 5939 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.547873 5939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548371 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548381 5939 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548384 5939 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548387 5939 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:29:47.552310 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548390 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548393 5939 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548396 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548398 5939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548401 5939 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548404 5939 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548407 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548410 5939 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548413 5939 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548415 5939 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548418 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548421 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548423 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548426 5939 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548429 5939 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548431 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548435 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548437 5939 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548440 5939 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:29:47.552958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548444 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548446 5939 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548450 5939 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548453 5939 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548457 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548459 5939 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548462 5939 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548466 5939 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548468 5939 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548471 5939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548478 5939 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548481 5939 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548483 5939 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548486 5939 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548489 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548491 5939 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548494 5939 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548497 5939 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548499 5939 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:29:47.553424 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548502 5939 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548505 5939 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548507 5939 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548510 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548513 5939 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548515 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548518 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548520 5939 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548523 5939 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548526 5939 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548529 5939 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548531 5939 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548534 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548536 5939 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548540 5939 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548543 5939 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548546 5939 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548548 5939 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548551 5939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548554 5939 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:29:47.553928 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548556 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548559 5939 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548561 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548565 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548568 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548571 5939 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548576 5939 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548581 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548584 5939 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548587 5939 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548590 5939 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548592 5939 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548595 5939 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548598 5939 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548600 5939 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548603 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548606 5939 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548608 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548611 5939 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548614 5939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:29:47.554412 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548616 5939 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548619 5939 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548621 5939 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548624 5939 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548626 5939 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548629 5939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548632 5939 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548634 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548637 5939 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548640 5939 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548642 5939 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548645 5939 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548647 5939 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548650 5939 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548652 5939 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548656 5939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548659 5939 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548661 5939 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548680 5939 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548683 5939 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:29:47.554899 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548685 5939 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548688 5939 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548690 5939 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.548693 5939 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548787 5939 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548794 5939 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548800 5939 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548805 5939 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548810 5939 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548813 5939 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548818 5939 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548822 5939 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548826 5939 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548829 5939 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548832 5939 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548835 5939 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548839 5939 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548842 5939 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548845 5939 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548848 5939 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548851 5939 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548854 5939 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548859 5939 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548862 5939 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:29:47.555370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548865 5939 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548868 5939 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548871 5939 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548877 5939 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548880 5939 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548883 5939 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548887 5939 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548891 5939 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548894 5939 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548897 5939 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548900 5939 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548903 5939 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548907 5939 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548911 5939 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548914 5939 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548917 5939 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548920 5939 flags.go:64] FLAG: --enable-server="true" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548923 5939 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548927 5939 flags.go:64] FLAG: --event-burst="100" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548931 5939 flags.go:64] FLAG: --event-qps="50" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548934 5939 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548937 5939 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548940 5939 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548944 5939 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:29:47.555996 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548947 5939 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548951 5939 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548954 5939 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548958 5939 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548961 5939 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548965 5939 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548968 5939 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548971 5939 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548974 5939 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548977 5939 flags.go:64] FLAG: --feature-gates="" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548981 5939 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548986 5939 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548989 5939 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548992 5939 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.548997 5939 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549000 5939 flags.go:64] FLAG: --help="false" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549003 5939 flags.go:64] FLAG: --hostname-override="ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549006 5939 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549009 5939 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549012 5939 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549016 5939 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549019 5939 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549022 5939 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:29:47.556811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549025 5939 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549028 5939 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549031 5939 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549034 5939 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549037 5939 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549041 5939 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549044 5939 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549047 5939 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549050 5939 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549053 5939 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549056 5939 flags.go:64] FLAG: --lock-file="" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549059 5939 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549062 5939 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549065 5939 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549070 5939 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549074 5939 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549077 5939 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549080 5939 flags.go:64] FLAG: --logging-format="text" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549082 5939 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549086 5939 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549091 5939 flags.go:64] FLAG: --manifest-url="" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549094 5939 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549098 5939 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549102 5939 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549106 5939 flags.go:64] FLAG: --max-pods="110" Oct 31 00:29:47.557382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549109 5939 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549113 5939 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549116 5939 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549119 5939 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549122 5939 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549125 5939 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549129 5939 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549137 5939 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549140 5939 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549143 5939 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549146 5939 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549149 5939 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549152 5939 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549155 5939 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549158 5939 flags.go:64] FLAG: --port="10250" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549161 5939 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549164 5939 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-05ccdfc004f20af88" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549168 5939 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549171 5939 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549174 5939 flags.go:64] FLAG: --register-node="true" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549177 5939 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549181 5939 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549184 5939 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549187 5939 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549190 5939 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549194 5939 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:29:47.558027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549197 5939 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549202 5939 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549205 5939 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549208 5939 flags.go:64] FLAG: --runonce="false" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549212 5939 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549215 5939 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549218 5939 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549221 5939 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549224 5939 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549227 5939 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549231 5939 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549234 5939 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549237 5939 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549240 5939 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549243 5939 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549246 5939 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549249 5939 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549252 5939 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549256 5939 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549261 5939 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549264 5939 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549267 5939 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549271 5939 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549274 5939 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549277 5939 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:29:47.558630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549279 5939 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549282 5939 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549285 5939 flags.go:64] FLAG: --v="2" Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549290 5939 flags.go:64] FLAG: --version="false" Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549294 5939 flags.go:64] FLAG: --vmodule="" Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549299 5939 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549303 5939 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549482 5939 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549489 5939 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549492 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549495 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549499 5939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549502 5939 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549504 5939 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549507 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549511 5939 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549515 5939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549519 5939 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549522 5939 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549526 5939 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549528 5939 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:29:47.559276 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549531 5939 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549534 5939 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549537 5939 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549540 5939 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549543 5939 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549545 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549548 5939 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549550 5939 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549553 5939 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549555 5939 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549558 5939 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549561 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549563 5939 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549566 5939 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549569 5939 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549571 5939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549574 5939 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549583 5939 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549586 5939 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549590 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:29:47.559878 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549593 5939 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549595 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549599 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549602 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549605 5939 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549607 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549610 5939 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549613 5939 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549615 5939 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549618 5939 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549621 5939 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549623 5939 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549626 5939 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549629 5939 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549631 5939 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549634 5939 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549636 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549639 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549641 5939 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549644 5939 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:29:47.560486 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549647 5939 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549649 5939 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549652 5939 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549654 5939 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549657 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549660 5939 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549676 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549679 5939 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549684 5939 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549687 5939 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549691 5939 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549694 5939 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549697 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549700 5939 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549703 5939 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549706 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549709 5939 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549711 5939 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549714 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:29:47.561573 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549716 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549719 5939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549721 5939 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549724 5939 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549726 5939 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549729 5939 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549732 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549734 5939 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549737 5939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549740 5939 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549742 5939 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549744 5939 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549747 5939 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549750 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549752 5939 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549755 5939 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549757 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549760 5939 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549763 5939 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549766 5939 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:29:47.562414 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549768 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:29:47.563347 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549771 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:29:47.563347 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549773 5939 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:29:47.563347 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549776 5939 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:29:47.563347 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549779 5939 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:29:47.563347 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549781 5939 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:29:47.563347 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549784 5939 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:29:47.563347 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549788 5939 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:29:47.563347 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549791 5939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:29:47.563347 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549793 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:29:47.563347 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549796 5939 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:29:47.563347 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549799 5939 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:29:47.563347 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549801 5939 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:29:47.563347 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.549826 5939 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:29:47.563347 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549867 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:29:47.563347 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549872 5939 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:29:47.563347 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549874 5939 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549877 5939 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549880 5939 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549883 5939 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549885 5939 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549888 5939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549890 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549893 5939 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549895 5939 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549898 5939 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549900 5939 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549903 5939 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549905 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549908 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549910 5939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549913 5939 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549915 5939 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549918 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549922 5939 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:29:47.563958 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549926 5939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549929 5939 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549933 5939 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549936 5939 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549940 5939 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549943 5939 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549945 5939 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549948 5939 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549950 5939 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549953 5939 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549955 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549958 5939 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549960 5939 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549963 5939 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549965 5939 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549968 5939 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549970 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549973 5939 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549975 5939 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549978 5939 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:29:47.564736 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549981 5939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549983 5939 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549986 5939 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549988 5939 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549991 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549993 5939 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549995 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.549998 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550000 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550003 5939 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550005 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550008 5939 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550010 5939 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550013 5939 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550015 5939 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550018 5939 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550022 5939 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550025 5939 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550028 5939 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550030 5939 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:29:47.565507 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550033 5939 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550035 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550038 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550041 5939 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550043 5939 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550046 5939 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550050 5939 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550054 5939 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550057 5939 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550060 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550062 5939 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550065 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550067 5939 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550071 5939 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550074 5939 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550077 5939 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550079 5939 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550082 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550084 5939 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:29:47.566237 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550087 5939 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550090 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550092 5939 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550095 5939 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550097 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550100 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550103 5939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550105 5939 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550107 5939 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550111 5939 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550114 5939 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550117 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550119 5939 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550122 5939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550124 5939 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550127 5939 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550129 5939 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550131 5939 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550134 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550137 5939 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:29:47.566890 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550141 5939 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550144 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550147 5939 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550150 5939 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550152 5939 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.550155 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.554419 5939 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.554435 5939 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.554452 5939 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.554457 5939 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554508 5939 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554514 5939 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554517 5939 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554521 5939 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554524 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554528 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554530 5939 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554535 5939 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554539 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:29:47.567560 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554542 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554546 5939 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554549 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554552 5939 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554555 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554558 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554560 5939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554563 5939 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554566 5939 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554568 5939 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554571 5939 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554574 5939 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554576 5939 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554579 5939 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554582 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554584 5939 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554587 5939 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554590 5939 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554593 5939 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554595 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:29:47.568302 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554598 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554601 5939 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554604 5939 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554606 5939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554609 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554611 5939 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554614 5939 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554616 5939 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554619 5939 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554622 5939 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554625 5939 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554628 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554630 5939 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554633 5939 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554637 5939 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554640 5939 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554644 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554648 5939 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554651 5939 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:29:47.569065 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554653 5939 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554656 5939 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554659 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554662 5939 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554678 5939 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554683 5939 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554686 5939 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554689 5939 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554692 5939 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554694 5939 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554697 5939 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554700 5939 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554702 5939 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554705 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554707 5939 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554710 5939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554713 5939 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554716 5939 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554718 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554721 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:29:47.569830 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554723 5939 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554726 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554728 5939 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554731 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554733 5939 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554736 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554739 5939 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554742 5939 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554744 5939 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554747 5939 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554750 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554754 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554756 5939 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554759 5939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554762 5939 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554764 5939 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554767 5939 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554769 5939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554772 5939 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:29:47.570541 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554774 5939 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554777 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554780 5939 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554782 5939 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554785 5939 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554787 5939 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554790 5939 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554792 5939 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554795 5939 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554797 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554800 5939 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554803 5939 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554806 5939 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554808 5939 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554811 5939 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554813 5939 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554816 5939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554819 5939 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:29:47.571230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554822 5939 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:29:47.571990 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.554848 5939 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:29:47.571990 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.554995 5939 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:29:47.571990 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555001 5939 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:29:47.571990 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555005 5939 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:29:47.571990 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555007 5939 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:29:47.571990 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555010 5939 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:29:47.571990 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555016 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:29:47.571990 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555019 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:29:47.571990 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555022 5939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:29:47.571990 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555025 5939 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:29:47.571990 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555028 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:29:47.571990 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555030 5939 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:29:47.571990 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555034 5939 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:29:47.571990 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555038 5939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:29:47.571990 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555041 5939 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555044 5939 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555046 5939 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555049 5939 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555052 5939 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555054 5939 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555057 5939 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555060 5939 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555062 5939 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555065 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555068 5939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555070 5939 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555073 5939 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555076 5939 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555079 5939 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555081 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555084 5939 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555087 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555089 5939 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:29:47.572554 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555092 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555095 5939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555097 5939 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555100 5939 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555102 5939 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555105 5939 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555108 5939 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555111 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555114 5939 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555116 5939 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555119 5939 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555121 5939 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555124 5939 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555127 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555129 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555132 5939 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555134 5939 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555137 5939 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555139 5939 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555144 5939 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:29:47.573272 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555147 5939 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555149 5939 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555152 5939 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555154 5939 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555157 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555159 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555162 5939 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555164 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555167 5939 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555170 5939 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555173 5939 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555175 5939 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555178 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555180 5939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555183 5939 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555185 5939 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555188 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555190 5939 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555193 5939 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555196 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:29:47.574114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555198 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555201 5939 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555205 5939 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555209 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555212 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555214 5939 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555216 5939 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555219 5939 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555222 5939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555225 5939 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555227 5939 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555232 5939 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555235 5939 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555237 5939 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555257 5939 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555261 5939 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555264 5939 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555267 5939 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555270 5939 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:29:47.574871 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555272 5939 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:29:47.575434 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555275 5939 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:29:47.575434 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555277 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:29:47.575434 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555280 5939 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:29:47.575434 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555282 5939 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:29:47.575434 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555285 5939 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:29:47.575434 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555288 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:29:47.575434 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555290 5939 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:29:47.575434 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555293 5939 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:29:47.575434 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555295 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:29:47.575434 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555298 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:29:47.575434 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555300 5939 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:29:47.575434 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555303 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:29:47.575434 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555306 5939 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:29:47.575434 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555309 5939 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:29:47.575434 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.555333 5939 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:29:47.575434 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555364 5939 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555369 5939 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555372 5939 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555374 5939 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555377 5939 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555380 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555383 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555386 5939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555390 5939 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555392 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555395 5939 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555399 5939 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555403 5939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555406 5939 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555408 5939 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555411 5939 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555414 5939 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555416 5939 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555419 5939 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:29:47.575992 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555421 5939 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555424 5939 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555426 5939 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555429 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555431 5939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555434 5939 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555437 5939 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555439 5939 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555442 5939 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555444 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555447 5939 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555450 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555453 5939 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555456 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555458 5939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555462 5939 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555465 5939 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555467 5939 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555470 5939 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:29:47.576569 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555473 5939 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555476 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555479 5939 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555482 5939 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555484 5939 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555487 5939 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555490 5939 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555494 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555496 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555499 5939 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555501 5939 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555504 5939 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555506 5939 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555509 5939 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555511 5939 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555514 5939 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555516 5939 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555519 5939 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555521 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555524 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:29:47.577114 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555526 5939 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555529 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555532 5939 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555534 5939 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555537 5939 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555540 5939 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555542 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555545 5939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555549 5939 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555551 5939 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555554 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555556 5939 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555559 5939 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555562 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555566 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555568 5939 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555572 5939 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555576 5939 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555579 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:29:47.577727 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555582 5939 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555585 5939 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555588 5939 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555591 5939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555594 5939 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555597 5939 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555601 5939 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555603 5939 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555606 5939 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555610 5939 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555612 5939 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555615 5939 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555618 5939 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555621 5939 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555623 5939 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555626 5939 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555628 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555631 5939 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555634 5939 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555636 5939 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:29:47.578349 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555639 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:29:47.579008 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555645 5939 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:29:47.579008 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555648 5939 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:29:47.579008 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555650 5939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:29:47.579008 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555653 5939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:29:47.579008 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555655 5939 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:29:47.579008 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555659 5939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:29:47.579008 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555676 5939 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:29:47.579008 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:29:47.555680 5939 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:29:47.579008 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.555738 5939 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:29:47.579008 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.556769 5939 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 00:29:47.579008 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.556823 5939 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 00:29:47.579008 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.556997 5939 server.go:1014] "Starting client certificate rotation" Oct 31 00:29:47.579008 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.557143 5939 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:29:47.579008 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.557192 5939 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:23:35 +0000 UTC" deadline="2028-04-28 21:17:58.331775884 +0000 UTC" Oct 31 00:29:47.579008 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.557218 5939 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13100h48m10.774560019s" Oct 31 00:29:47.579008 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.557472 5939 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:29:47.579444 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.558516 5939 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:29:47.579444 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.559923 5939 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:29:47.579444 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.563105 5939 log.go:25] "Validated CRI v1 image API" Oct 31 00:29:47.579444 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.576760 5939 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:29:47.579555 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.579448 5939 fs.go:135] Filesystem UUIDs: map[567967fd-b706-463a-98d4-9a132fb073db:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fd096270-1e24-470b-bc7a-fa03b10dfec1:/dev/nvme0n1p4] Oct 31 00:29:47.579994 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.579469 5939 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1389b5756470cd0ebf6e3e93cfb8814ab7e582ca290a67e548ceffba753a1933/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1389b5756470cd0ebf6e3e93cfb8814ab7e582ca290a67e548ceffba753a1933/userdata/shm major:0 minor:257 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f0fb6a0696bb35aba27a48e5b95aea97af5c07216c282ab7cd1925e39c6408c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f0fb6a0696bb35aba27a48e5b95aea97af5c07216c282ab7cd1925e39c6408c/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/290dfca05c359ae7d59ff7029b87f39951a55146fa46cfff9478e29183348f0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/290dfca05c359ae7d59ff7029b87f39951a55146fa46cfff9478e29183348f0e/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2dfc3f7ccf8877dde406b15a8d42f32ff79e2c4fe1235e03218ef60570730b5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2dfc3f7ccf8877dde406b15a8d42f32ff79e2c4fe1235e03218ef60570730b5e/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e23bab50c3ff188c8886ae680ea2ed80410934050277e12433468e15d3bbf07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e23bab50c3ff188c8886ae680ea2ed80410934050277e12433468e15d3bbf07/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34cac5d6a5a3bffaec384bbcac6431fd3ab274264163a27a4188dd9723401e3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34cac5d6a5a3bffaec384bbcac6431fd3ab274264163a27a4188dd9723401e3a/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/371eb6cd0849d585a4ab3d88b1ea5b40529fe1f3d8cbb61047ae753cd5844dba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/371eb6cd0849d585a4ab3d88b1ea5b40529fe1f3d8cbb61047ae753cd5844dba/userdata/shm major:0 minor:491 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3de12aaf77de82f2be7323f69e46b04a88db46512e6a234722126ffd81fb1775/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3de12aaf77de82f2be7323f69e46b04a88db46512e6a234722126ffd81fb1775/userdata/shm major:0 minor:364 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/44dbf45ebf1a60401e63bee5572ce4fff1a74f91a4d83c6957d3940323784534/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/44dbf45ebf1a60401e63bee5572ce4fff1a74f91a4d83c6957d3940323784534/userdata/shm major:0 minor:262 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e3112303462bb8e25b0958bc4904b479a86d527e72103b056d912e1df983c7c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e3112303462bb8e25b0958bc4904b479a86d527e72103b056d912e1df983c7c/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6585c48b6b476cf57dd29c701e10242c6998dcf16f086da180f099973f1cdc9c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6585c48b6b476cf57dd29c701e10242c6998dcf16f086da180f099973f1cdc9c/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80ffc814accc550d820b6b4f4723f239b1bf007502c43055fae6a38b2b7b1903/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80ffc814accc550d820b6b4f4723f239b1bf007502c43055fae6a38b2b7b1903/userdata/shm major:0 minor:391 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8306d8cae5a33092875f586bcde7ed1706c565c930d21e4863f49018b186e564/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8306d8cae5a33092875f586bcde7ed1706c565c930d21e4863f49018b186e564/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91572e8ab1e6067a5d2f4f9a1e81b1859c2d37d3048dda1a2a4aa345af7b7b2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91572e8ab1e6067a5d2f4f9a1e81b1859c2d37d3048dda1a2a4aa345af7b7b2a/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a7aad1aa8e94fa8a746240dcea56145b51be9f17b836a8a1ec4e3860c9a4e39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a7aad1aa8e94fa8a746240dcea56145b51be9f17b836a8a1ec4e3860c9a4e39/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/acaf343745ccfd10174cdb916b6488c6fe4fb0428d40402e6c863d3e03a9f596/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/acaf343745ccfd10174cdb916b6488c6fe4fb0428d40402e6c863d3e03a9f596/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b69fd75ea1b90792846d487eba2508eb6db2af1ee90053514322be191f96b622/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b69fd75ea1b90792846d487eba2508eb6db2af1ee90053514322be191f96b622/userdata/shm major:0 minor:488 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c45165966107cab5eea0f3a82ed97e285ea82de8829878df0a678cee5634d91c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c45165966107cab5eea0f3a82ed97e285ea82de8829878df0a678cee5634d91c/userdata/shm major:0 minor:439 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5ea6728d5c08fe5bc5e5c62b3219740867b0e727c7faa521ad1ac74cfda5146/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5ea6728d5c08fe5bc5e5c62b3219740867b0e727c7faa521ad1ac74cfda5146/userdata/shm major:0 minor:493 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cde84019a225b3e17600718d17090867122a504362b61485753b93ef0f2767d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cde84019a225b3e17600718d17090867122a504362b61485753b93ef0f2767d9/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d99f9a3644d63d499fbc5ebf8b9687ca9faec6c7acdc28dbf6758af61e812b72/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d99f9a3644d63d499fbc5ebf8b9687ca9faec6c7acdc28dbf6758af61e812b72/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e768af5000c0dcee47d0b3b7dafcfd78487ca2a82c5a03555e371bb361caa13d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e768af5000c0dcee47d0b3b7dafcfd78487ca2a82c5a03555e371bb361caa13d/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb1837caaa867184184cb737ed2ca7fbe06a6b484638b74d5fd579b1b5b896fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb1837caaa867184184cb737ed2ca7fbe06a6b484638b74d5fd579b1b5b896fd/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed0544758ce9298b0c6d9f2ab0d50e637724e41dddb40c4cb5380dec8282de29/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed0544758ce9298b0c6d9f2ab0d50e637724e41dddb40c4cb5380dec8282de29/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f40daf736160f81c37cdafce2c0fed75113a79f2955f39de17ecf4df7ed44f82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f40daf736160f81c37cdafce2c0fed75113a79f2955f39de17ecf4df7ed44f82/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1db4e02b-1e7a-4361-994e-f30507dbf0e7/volumes/kubernetes.io~projected/kube-api-access-gcmn7:{mountpoint:/var/lib/kubelet/pods/1db4e02b-1e7a-4361-994e-f30507dbf0e7/volumes/kubernetes.io~projected/kube-api-access-gcmn7 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/266a13d4-3f45-44d6-ac0e-3e171ce4e928/volumes/kubernetes.io~projected/kube-api-access-29dqz:{mountpoint:/var/lib/kubelet/pods/266a13d4-3f45-44d6-ac0e-3e171ce4e928/volumes/kubernetes.io~projected/kube-api-access-29dqz major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/266a13d4-3f45-44d6-ac0e-3e171ce4e928/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/266a13d4-3f45-44d6-ac0e-3e171ce4e928/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3073e91a-8363-4d2a-8774-635302d4b040/volumes/kubernetes.io~projected/kube-api-access-4wvxm:{mountpoint:/var/lib/kubelet/pods/3073e91a-8363-4d2a-8774-635302d4b040/volumes/kubernetes.io~projected/kube-api-access-4wvxm major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37dd17e5-ba62-4f95-bfc7-05b132104e44/volumes/kubernetes.io~projected/kube-api-access-wvdnb:{mountpoint:/var/lib/kubelet/pods/37dd17e5-ba62-4f95-bfc7-05b132104e44/volumes/kubernetes.io~projected/kube-api-access-wvdnb major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37dd17e5-ba62-4f95-bfc7-05b132104e44/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/37dd17e5-ba62-4f95-bfc7-05b132104e44/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40c70642-d50d-4f92-8ca1-10015c1a40cd/volumes/kubernetes.io~projected/kube-api-access-qcx7m:{mountpoint:/var/lib/kubelet/pods/40c70642-d50d-4f92-8ca1-10015c1a40cd/volumes/kubernetes.io~projected/kube-api-access-qcx7m major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40c70642-d50d-4f92-8ca1-10015c1a40cd/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/40c70642-d50d-4f92-8ca1-10015c1a40cd/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40c70642-d50d-4f92-8ca1-10015c1a40cd/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/40c70642-d50d-4f92-8ca1-10015c1a40cd/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51dbc54b-218b-4274-9ebb-4cdbab400d22/volumes/kubernetes.io~projected/kube-api-access-h6gnm:{mountpoint:/var/lib/kubelet/pods/51dbc54b-218b-4274-9ebb-4cdbab400d22/volumes/kubernetes.io~projected/kube-api-access-h6gnm major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51dbc54b-218b-4274-9ebb-4cdbab400d22/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/51dbc54b-218b-4274-9ebb-4cdbab400d22/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57decfb3-58bb-4b9c-b834-205594071bdf/volumes/kubernetes.io~projected/kube-api-access-v6j5l:{mountpoint:/var/lib/kubelet/pods/57decfb3-58bb-4b9c-b834-205594071bdf/volumes/kubernetes.io~projected/kube-api-access-v6j5l major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57decfb3-58bb-4b9c-b834-205594071bdf/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/57decfb3-58bb-4b9c-b834-205594071bdf/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede/volumes/kubernetes.io~projected/kube-api-access-j8cjz:{mountpoint:/var/lib/kubelet/pods/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede/volumes/kubernetes.io~projected/kube-api-access-j8cjz major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6dca6aeb-205b-4634-9216-a98ba3957025/volumes/kubernetes.io~projected/kube-api-access-26png:{mountpoint:/var/lib/kubelet/pods/6dca6aeb-205b-4634-9216-a98ba3957025/volumes/kubernetes.io~projected/kube-api-access-26png major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f86ab44-a957-4e07-8bdf-5a8f813f27d4/volumes/kubernetes.io~projected/kube-api-access-f7lkc:{mountpoint:/var/lib/kubelet/pods/6f86ab44-a957-4e07-8bdf-5a8f813f27d4/volumes/kubernetes.io~projected/kube-api-access-f7lkc major:0 minor:490 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79b89acd-a759-4036-bd91-16ee9ad9e288/volumes/kubernetes.io~projected/kube-api-access-mdn2m:{mountpoint:/var/lib/kubelet/pods/79b89acd-a759-4036-bd91-16ee9ad9e288/volumes/kubernetes.io~projected/kube-api-access-mdn2m major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~projected/kube-api-access-bmzx8:{mountpoint:/var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~projected/kube-api-access-bmzx8 major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d06594d-071d-47bb-ade5-16298fd91e2c/volumes/kubernetes.io~projected/kube-api-access-bthrt:{mountpoint:/var/lib/kubelet/pods/8d06594d-071d-47bb-ade5-16298fd91e2c/volumes/kubernetes.io~projected/kube-api-access-bthrt major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d06594d-071d-47bb-ade5-16298fd91e2c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8d06594d-071d-47bb-ade5-16298fd91e2c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d06594d-071d-47bb-ade5-16298fd91e2c/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/8d06594d-071d-47bb-ade5-16298fd91e2c/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91208431-95bb-4e24-ab4b-e0f80f0d6207/volumes/kubernetes.io~projected/kube-api-access-4mfhm:{mountpoint:/var/lib/kubelet/pods/91208431-95bb-4e24-ab4b-e0f80f0d6207/volumes/kubernetes.io~projected/kube-api-access-4mfhm major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91208431-95bb-4e24-ab4b-e0f80f0d6207/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/91208431-95bb-4e24-ab4b-e0f80f0d6207/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91208431-95bb-4e24-ab4b-e0f80f0d6207/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/91208431-95bb-4e24-ab4b-e0f80f0d6207/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4677743-13cb-4a3a-b1df-8bd6baef94f4/volumes/kubernetes.io~projected/kube-api-access-22g5c:{mountpoint:/var/lib/kubelet/pods/a4677743-13cb-4a3a-b1df-8bd6baef94f4/volumes/kubernetes.io~projected/kube-api-access-22g5c major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4677743-13cb-4a3a-b1df-8bd6baef94f4/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a4677743-13cb-4a3a-b1df-8bd6baef94f4/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4677743-13cb-4a3a-b1df-8bd6baef94f4/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/a4677743-13cb-4a3a-b1df-8bd6baef94f4/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2eb2a42-2573-414d-8516-aac46f430550/volumes/kubernetes.io~projected/kube-api-access-xwjlt:{mountpoint:/var/lib/kubelet/pods/b2eb2a42-2573-414d-8516-aac46f430550/volumes/kubernetes.io~projected/kube-api-access-xwjlt major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2eb2a42-2573-414d-8516-aac46f430550/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/b2eb2a42-2573-414d-8516-aac46f430550/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2eb2a42-2573-414d-8516-aac46f430550/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/b2eb2a42-2573-414d-8516-aac46f430550/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2eb2a42-2573-414d-8516-aac46f430550/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/b2eb2a42-2573-414d-8516-aac46f430550/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba178af9-0f61-4c7b-a5ab-36e772495016/volumes/kubernetes.io~projected/kube-api-access-b8mpp:{mountpoint:/var/lib/kubelet/pods/ba178af9-0f61-4c7b-a5ab-36e772495016/volumes/kubernetes.io~projected/kube-api-access-b8mpp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba178af9-0f61-4c7b-a5ab-36e772495016/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/ba178af9-0f61-4c7b-a5ab-36e772495016/volumes/kubernetes.io~secret/metrics-certs major:0 minor:483 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volume-subpaths/web-config/alertmanager/9 major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volume-subpaths/web-config/config-reloader/6 major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~empty-dir/config-out major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~projected/kube-api-access-sc9ss:{mountpoint:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~projected/kube-api-access-sc9ss major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~projected/tls-assets major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/config-volume major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/web-config major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c892baf2-ce51-4c47-8d79-f75f4f50c09d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c892baf2-ce51-4c47-8d79-f75f4f50c09d/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0/volumes/kubernetes.io~projected/kube-api-access-q8dl6:{mountpoint:/var/lib/kubelet/pods/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0/volumes/kubernetes.io~projected/kube-api-access-q8dl6 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7b81351-0c71-49b6-bd86-e48483b8b555/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e7b81351-0c71-49b6-bd86-e48483b8b555/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:487 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/volumes/kubernetes.io~projected/kube-api-access-xh2g8:{mountpoint:/var/lib/kubelet/pods/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/volumes/kubernetes.io~projected/kube-api-access-xh2g8 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa/volumes/kubernetes.io~projected/kube-api-access-lr2lm:{mountpoint:/var/lib/kubelet/pods/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa/volumes/kubernetes.io~projected/kube-api-access-lr2lm major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volume-subpaths/web-config/config-reloader/2 major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volume-subpaths/web-config/prometheus/15 major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~empty-dir/config-out major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~projected/kube-api-access-tbxs7:{mountpoint:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~projected/kube-api-access-tbxs7 major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~projected/tls-assets major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/config major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:433 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/web-config major:0 minor:436 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/cf5270e71b78cb9ae6ead38e12d625a30ef0f2e389853c5a56b48ce0b1d47a6a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/cf37324c9c99a7b6bcb102b5ee5547f4651508a0bbc9aa4def0125e5cb60acb3/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7c0fe4c70d168a158b6753902d5a3e8be6ca351c5132838b034f7c5d38efcbbc/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/be7219bc68f26043dfabc176bf0d12f3b17b22daa13254e1aa06fb3c7395a50f/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/18dc540ef2b84c0b9ea1759346846eb9332acef951ad779e562fb80746d1e141/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/6b9ea1ffa93caeb1b094c3e3c1297ed7f757163cbe9e83d6d6915820f664eecb/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/7e0a427f30a4b38d1bd4f4e945c440733089660cd386509fffad749655786641/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/39fc5d5fc715d1ea5c4241ebcda63c632b315e21537dcbf73d6eb4f98f87759b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/63fd0b0612ff201de3b2106931e63e5152e4041ee744096792de1f87e295b7d1/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/338cf7cd883b77252cb0613b14c8a24580ccebfaecc870025dbc8e95a222d77d/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/e24067a906a8821c0abe20400f925a35f13edb793bc0d8a9b0f10d694bb3a6ed/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/ba35c8afc4a7894ec98ec305b6f95e4cc4a21f501e64e32412e1b4db75997b9a/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c1370e37f6084ebba5e3fd8ee993380e7c57e99fe7a8499cc50bf5b15dfe3232/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1cb6a80f7cbc0ad0983225d20997c49da2d9dce1415b5a6cc193319ceecc61e7/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/0b0a0283bb1806f2ac5dec19f81687dbc80df92c30efbafbedbecadcf2cbef14/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/72305e78db929e85c8be3d07761b896013c47a2ca731245d5f4a5d8bcbec0ea1/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/1da72ff5db0627ba08aea760836b8c13cff39fffa02ea34a54411faeb23c2fb2/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/77b238fb6c742ee684da801d8339c59b74e94d9831fa762ad497547371dd9f0f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/588fb7a7feafd0b871dfb87d637e0a2251b513481e31642161a37bcbb016f499/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/5b84e53bae4d5cc326099d5c27b1df76c16ec0622abe57bb201edea96534809f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/1963ff3d49ce4ce525ac1754a9ceed81e3c40e29546ffae8147bd5ad470063cf/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/ce33d0a777cddab188a4983fed76fbf2afc1cc2e1aee91caa159f02c0e80f696/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/5697a36d44f8e6b015ca08d57ae1a32c1ac799991dbc4834d14da113ee36a17d/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/44c886f90eb9b356e5edbd390fdb64c86f375cc7ad1dc2dee2c136a5429ade24/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/701a6069e8f6721d4c1fae73955705c70baf77b54766ef6977a9d24f384be8cd/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/59f656e79de0458f11a97805e4ee3fe72475c42fe66da052e5db929f3547aaea/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/36e79e452b69b2ff2a677fa0aea826da24fbd1568bf6712eefddf48fea1a3599/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/da2917122ac61a6c0af9ba1a6ea1e58c77d6bc34622dd940e1f4eea9c203f089/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/e29b09b16430916ad76ba4bb9b284b8346d2e569794658b8e6769bd21989913a/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/1321b7bf3f6fc534c9a44ba90a9f892f3b0f0367ba8334fde604da5b3bc35074/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/2ba8abafa321316613c46b911b98adc286ab1e8844f47d634fda29a504b13021/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/c340ff09b0d4000f739a87ea533eb6d7c66aae2a12b9c6fcd0e9d957f4e04245/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/d24ff4087ca2883b8b8d56806c64d67ec6347f7e784d3504d76cfc557722428e/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/f595a4e8775be105918eadf3d3522f393c61f0984c83876ca5dff6e47682360d/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/8ded4e6c3955ab9141b5e5a92982e5ca25ae95264239a0c4faa6559c343e0d95/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/e19bd87087271374a8e9264c20095c9b6ae893163185c0fbb1fa2eb3c9bd3ad1/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/8e115a0bae221723677e16d454f5ac281d6cc10fbaeba889f003f990df91bcef/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/ad22fb3d21d7ad15bf95cd082384a4e96826b7fe2d54aaaa1cad3d958bd01244/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/ea3d895289695af0e68806020652073bfca9a7333dc099b1528ac252267046f2/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/3bb3e9ae7dbfc1040702fb82436895482bb5aaa9f8d7a30fbfd8451cc8da851a/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/51953867de44a85e7c9ca8f7c15e40f184e7c7b605d8d7e4369570e58784a5ac/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/ac935a04319e56667f49ec270adc69b0dbfe3824dbbfa15b45ac6e7cf4e6d8b2/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/dfa889aae31070f1c75ee2ed96a1295ad69247fb8e82867bbf902cc5195b3a91/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/480761ebddd9ded1cca189fb05b82cb0de4eaeee913ff46e5d15640cfc6e7ffa/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/aeea39d6f3b1fbc9ab45e0c16a9cf55ae4513b7cb6a097fca6b6f1d16aba1e6a/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/db1f0b1f58ed56b9c7ff467b434f897ddc385cf56604a9ebeccd9cf4fad0d8ff/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/e24017499e38e9d4ab8de00734322fd2610381f3abfb2dd0467a5d4fb1454d7b/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/2ada708bdd2f99152de4bc5396a3da5c674003569733e0fb3a7a0c621acc7ccf/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/2c5a3a9e4c825551e60e581cb6eccea4723a33857318f7db1331289649f9bc36/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/52255330ed9f6b30012da1e10c2d546b499aad01422d372efb618c2ee98b7d3d/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/b3951c3479b86c140713554d378162eb9625efd51c065c1a5a03d8d0ad912933/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/5b2ddb58f1a6f05325e05689615795cda1a7a658cc512d124637819e1341bc38/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/c4498ca7608dcb62989e151fd5ba9ce954d3300261e4bbf61bd7aff10780a219/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/555bb63f9a8851361934423be08df53ce1bdbc17505ceb10fab98d32578349fb/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/115875d0c566fad2a494d2d4f77a417b8acf8f4c186b027bc8299afba2806471/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/ae0f17b5be3b4af3245493c83db1f460b2ea7503948b57345dec216a3539bcd8/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/6bcf8ffb2251b6ace2456b1aa2d133a0170f4386c3e46e03ce14527f8d5e7ea2/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/998a74b69b08426ed73dac224efb5ad544d7a2523c640da9db760001e777372b/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/ca0aa3c2b2d641d39e249a177000572c90643ad36bb2a91548d233d7bdee778b/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/37b2e39db35b7d00a6c5eb1528b68d8f3e45297bec5c1309b329762ddba46ab6/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/637dc78caa29d40a287d5e619e48df0b602b534f5be753f207a63625cd109983/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/b7ca1be24820cd4fec102231a4dbf4954b16db418b9cab6c999d09d3d8d142cd/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/dd3ac284aeeec4a39c05f740ef7e775b34dbe704a5636a2aed5b7479e05c880f/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-457:{mountpoint:/var/lib/containers/storage/overlay/3c5a6d858ef691cec303482a1d90c0368ca2a3f68707ef63f55f24812ad095de/merged major:0 minor:457 fsType:overlay blockSize:0} overlay_0-463:{mountpoint:/var/lib/containers/storage/overlay/442212dd459a356d034c5ea3d0d97f6072b313c80556cc244455cf292ce2a860/merged major:0 minor:463 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/cf903f73d5691e8abcbb42b526a2375c7067d09f7e543de553fd4f6585d2aeac/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/2d58d1cd2ed8120b69b4c0704663f16d19121a35630432fb85146b38f63e7e33/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-478:{mountpoint:/var/lib/containers/storage/overlay/2ee58e4e18f8bc713981574b8917011b5defb8d62261c236c11bd97bc6d4bd35/merged major:0 minor:478 fsType:overlay blockSize:0} overlay_0-495:{mountpoint:/var/lib/containers/storage/overlay/d6a8351791c8630e3c8318f63e57237c20bddcd0e2c94c8ea7eee56547658b26/merged major:0 minor:495 fsType:overlay blockSize:0} overlay_0-497:{mountpoint:/var/lib/containers/storage/overlay/822e309aa490f468c7ce5b8c33b2ae72a60a40241428394acbace8f4aa8f59d6/merged major:0 minor:497 fsType:overlay blockSize:0} overlay_0-499:{mountpoint:/var/lib/containers/storage/overlay/c403f83ba2f3e6f9fe4713724b5d7672fceb80c15cba6c8bc629e501cbb58b79/merged major:0 minor:499 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/ec73ae63f83267aa999a630807175abd313535bbc5ee20ec0368799490731f37/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/489eb3fe4c57ec0cdc2307a0b7220ee63c3d670e82f1878fb035930f45004be7/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-509:{mountpoint:/var/lib/containers/storage/overlay/14a4867358c2ae674146205cf53ccb5e800b426e2f663f13f27e903f95ad8704/merged major:0 minor:509 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/689dc6fdcabfbbee02197b238a075a6d91ac235ea34cac46445df2eefe8d0341/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-514:{mountpoint:/var/lib/containers/storage/overlay/d5e340bf5215e71937e9816981bf3df642de8535f9e95f6e7757fce6db227a7c/merged major:0 minor:514 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/99986647f8565e56fb6788ce0e50ee9c7fd95cec8fef8c0752d97f30e1b7747e/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/68190cb5aff874b6b5ed91ac123e042f284b48e87be98ea6e328be0b524c53d1/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/7cbc395d2e6a273d782d0f70a468733f5fe19d914fd76ecbbaa702598e094b6f/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/58f61d9e0a57738da8157a9b4fbdc879fb1922d9f9c76704c67839043e603cee/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/63e5c93acf651a8410102d80a3522e2543153ec66095180b55d3c5450b1ba01d/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/d7b7c38cdd192a848c39a3f35a9604fec29f2b71e325f1e8f59458a036d4431d/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/ab223c920265a8b3b8a6f8d871154b5d076d5ff8c98f1fbb67a6c7fc6ba3339d/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/963a018330d4cfb57bbf47543a58a2aa5ffd6e6762fe2c0a217af4712ec1da51/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/77bec3e2b2ba837bafbb13ec880233637bde1fded86bd0f0121150c491e739c3/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 00:29:47.590405 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.589957 5939 manager.go:217] Machine: {Timestamp:2026-10-31 00:29:47.589453501 +0000 UTC m=+0.078728673 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200988 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ea9daa5c10b1d293b1156d632f987 SystemUUID:ec2ea9da-a5c1-0b1d-293b-1156d632f987 BootID:882d6525-2724-4f0e-a603-d7801e0111ec Filesystems:[{Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/volumes/kubernetes.io~projected/kube-api-access-xh2g8 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/37dd17e5-ba62-4f95-bfc7-05b132104e44/volumes/kubernetes.io~projected/kube-api-access-wvdnb DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~projected/kube-api-access-bmzx8 DeviceMajor:0 DeviceMinor:315 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2eb2a42-2573-414d-8516-aac46f430550/volumes/kubernetes.io~projected/kube-api-access-xwjlt DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/266a13d4-3f45-44d6-ac0e-3e171ce4e928/volumes/kubernetes.io~projected/kube-api-access-29dqz DeviceMajor:0 DeviceMinor:370 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e3112303462bb8e25b0958bc4904b479a86d527e72103b056d912e1df983c7c/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:303 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:433 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:436 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2eb2a42-2573-414d-8516-aac46f430550/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:351 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34cac5d6a5a3bffaec384bbcac6431fd3ab274264163a27a4188dd9723401e3a/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3de12aaf77de82f2be7323f69e46b04a88db46512e6a234722126ffd81fb1775/userdata/shm DeviceMajor:0 DeviceMinor:364 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~projected/kube-api-access-tbxs7 DeviceMajor:0 DeviceMinor:438 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/51dbc54b-218b-4274-9ebb-4cdbab400d22/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/57decfb3-58bb-4b9c-b834-205594071bdf/volumes/kubernetes.io~projected/kube-api-access-v6j5l DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a4677743-13cb-4a3a-b1df-8bd6baef94f4/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:429 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e7b81351-0c71-49b6-bd86-e48483b8b555/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:487 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b69fd75ea1b90792846d487eba2508eb6db2af1ee90053514322be191f96b622/userdata/shm DeviceMajor:0 DeviceMinor:488 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/371eb6cd0849d585a4ab3d88b1ea5b40529fe1f3d8cbb61047ae753cd5844dba/userdata/shm DeviceMajor:0 DeviceMinor:491 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/79b89acd-a759-4036-bd91-16ee9ad9e288/volumes/kubernetes.io~projected/kube-api-access-mdn2m DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb1837caaa867184184cb737ed2ca7fbe06a6b484638b74d5fd579b1b5b896fd/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b2eb2a42-2573-414d-8516-aac46f430550/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:352 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/1db4e02b-1e7a-4361-994e-f30507dbf0e7/volumes/kubernetes.io~projected/kube-api-access-gcmn7 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40c70642-d50d-4f92-8ca1-10015c1a40cd/volumes/kubernetes.io~projected/kube-api-access-qcx7m DeviceMajor:0 DeviceMinor:248 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:313 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:432 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57decfb3-58bb-4b9c-b834-205594071bdf/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:309 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a7aad1aa8e94fa8a746240dcea56145b51be9f17b836a8a1ec4e3860c9a4e39/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4677743-13cb-4a3a-b1df-8bd6baef94f4/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:311 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40c70642-d50d-4f92-8ca1-10015c1a40cd/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:247 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f40daf736160f81c37cdafce2c0fed75113a79f2955f39de17ecf4df7ed44f82/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80ffc814accc550d820b6b4f4723f239b1bf007502c43055fae6a38b2b7b1903/userdata/shm DeviceMajor:0 DeviceMinor:391 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede/volumes/kubernetes.io~projected/kube-api-access-j8cjz DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8306d8cae5a33092875f586bcde7ed1706c565c930d21e4863f49018b186e564/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91572e8ab1e6067a5d2f4f9a1e81b1859c2d37d3048dda1a2a4aa345af7b7b2a/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8d06594d-071d-47bb-ade5-16298fd91e2c/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba178af9-0f61-4c7b-a5ab-36e772495016/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:483 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0/volumes/kubernetes.io~projected/kube-api-access-q8dl6 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2dfc3f7ccf8877dde406b15a8d42f32ff79e2c4fe1235e03218ef60570730b5e/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d06594d-071d-47bb-ade5-16298fd91e2c/volumes/kubernetes.io~projected/kube-api-access-bthrt DeviceMajor:0 DeviceMinor:256 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~projected/kube-api-access-sc9ss DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6dca6aeb-205b-4634-9216-a98ba3957025/volumes/kubernetes.io~projected/kube-api-access-26png DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cde84019a225b3e17600718d17090867122a504362b61485753b93ef0f2767d9/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f0fb6a0696bb35aba27a48e5b95aea97af5c07216c282ab7cd1925e39c6408c/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a4677743-13cb-4a3a-b1df-8bd6baef94f4/volumes/kubernetes.io~projected/kube-api-access-22g5c DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e23bab50c3ff188c8886ae680ea2ed80410934050277e12433468e15d3bbf07/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/290dfca05c359ae7d59ff7029b87f39951a55146fa46cfff9478e29183348f0e/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40c70642-d50d-4f92-8ca1-10015c1a40cd/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:243 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/44dbf45ebf1a60401e63bee5572ce4fff1a74f91a4d83c6957d3940323784534/userdata/shm DeviceMajor:0 DeviceMinor:262 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/37dd17e5-ba62-4f95-bfc7-05b132104e44/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c45165966107cab5eea0f3a82ed97e285ea82de8829878df0a678cee5634d91c/userdata/shm DeviceMajor:0 DeviceMinor:439 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5ea6728d5c08fe5bc5e5c62b3219740867b0e727c7faa521ad1ac74cfda5146/userdata/shm DeviceMajor:0 DeviceMinor:493 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-509 DeviceMajor:0 DeviceMinor:509 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:436 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-497 DeviceMajor:0 DeviceMinor:497 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:436 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-478 DeviceMajor:0 DeviceMinor:478 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/91208431-95bb-4e24-ab4b-e0f80f0d6207/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/91208431-95bb-4e24-ab4b-e0f80f0d6207/volumes/kubernetes.io~projected/kube-api-access-4mfhm DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c892baf2-ce51-4c47-8d79-f75f4f50c09d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa/volumes/kubernetes.io~projected/kube-api-access-lr2lm DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d99f9a3644d63d499fbc5ebf8b9687ca9faec6c7acdc28dbf6758af61e812b72/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:430 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:434 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6585c48b6b476cf57dd29c701e10242c6998dcf16f086da180f099973f1cdc9c/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1389b5756470cd0ebf6e3e93cfb8814ab7e582ca290a67e548ceffba753a1933/userdata/shm DeviceMajor:0 DeviceMinor:257 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2eb2a42-2573-414d-8516-aac46f430550/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:353 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/266a13d4-3f45-44d6-ac0e-3e171ce4e928/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:374 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:431 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:435 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:436 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-457 DeviceMajor:0 DeviceMinor:457 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:437 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-499 DeviceMajor:0 DeviceMinor:499 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e768af5000c0dcee47d0b3b7dafcfd78487ca2a82c5a03555e371bb361caa13d/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d06594d-071d-47bb-ade5-16298fd91e2c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/51dbc54b-218b-4274-9ebb-4cdbab400d22/volumes/kubernetes.io~projected/kube-api-access-h6gnm DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/91208431-95bb-4e24-ab4b-e0f80f0d6207/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-495 DeviceMajor:0 DeviceMinor:495 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba178af9-0f61-4c7b-a5ab-36e772495016/volumes/kubernetes.io~projected/kube-api-access-b8mpp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/acaf343745ccfd10174cdb916b6488c6fe4fb0428d40402e6c863d3e03a9f596/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-514 DeviceMajor:0 DeviceMinor:514 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/86e13c77-adcc-4f59-b1a4-93d090e2379e/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3073e91a-8363-4d2a-8774-635302d4b040/volumes/kubernetes.io~projected/kube-api-access-4wvxm DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:428 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-463 DeviceMajor:0 DeviceMinor:463 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:302 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:424 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed0544758ce9298b0c6d9f2ab0d50e637724e41dddb40c4cb5380dec8282de29/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:298 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f86ab44-a957-4e07-8bdf-5a8f813f27d4/volumes/kubernetes.io~projected/kube-api-access-f7lkc DeviceMajor:0 DeviceMinor:490 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1389b5756470cd0 MacAddress:f2:f1:66:d2:a1:93 Speed:10000 Mtu:8901} {Name:1f0fb6a0696bb35 MacAddress:32:e0:2c:9a:4e:85 Speed:10000 Mtu:8901} {Name:290dfca05c359ae MacAddress:fe:bc:05:26:a9:e2 Speed:10000 Mtu:8901} {Name:371eb6cd0849d58 MacAddress:2a:61:66:2d:94:c7 Speed:10000 Mtu:8901} {Name:3de12aaf77de82f MacAddress:1a:a7:fc:d2:58:0b Speed:10000 Mtu:8901} {Name:80ffc814accc550 MacAddress:02:d4:f4:ed:9a:69 Speed:10000 Mtu:8901} {Name:91572e8ab1e6067 MacAddress:fe:cf:42:16:02:19 Speed:10000 Mtu:8901} {Name:acaf343745ccfd1 MacAddress:f2:2d:49:dd:30:6d Speed:10000 Mtu:8901} {Name:b69fd75ea1b9079 MacAddress:ee:17:f9:f3:fe:60 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d5:7b:03:53 Speed:0 Mtu:9001} {Name:br-int MacAddress:a2:7a:5d:d5:05:93 Speed:0 Mtu:8901} {Name:c45165966107cab MacAddress:b6:ff:16:ca:41:95 Speed:10000 Mtu:8901} {Name:c5ea6728d5c08fe MacAddress:9e:7d:80:54:1a:26 Speed:10000 Mtu:8901} {Name:d99f9a3644d63d4 MacAddress:32:1f:be:c7:a1:bb Speed:10000 Mtu:8901} {Name:eb1837caaa86718 MacAddress:fa:b9:07:a0:b4:b8 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d5:7b:03:53 Speed:0 Mtu:9001} {Name:f40daf736160f81 MacAddress:46:59:ad:72:46:8a Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:16:83:c8:26:8b:68 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1a:09:e1:a2:a0:bc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:29:47.590405 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.590407 5939 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:29:47.590781 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.590453 5939 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:29:47.590813 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.590767 5939 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:29:47.590930 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.590794 5939 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-213.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:29:47.590971 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.590932 5939 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:29:47.590971 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.590940 5939 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:29:47.590971 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.590951 5939 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:29:47.590971 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.590964 5939 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:29:47.591099 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.591066 5939 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:29:47.591255 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.591245 5939 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:29:47.591368 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.591359 5939 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:29:47.591407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.591398 5939 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:29:47.591438 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.591412 5939 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:29:47.591438 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.591424 5939 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:29:47.591438 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.591433 5939 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:29:47.591542 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.591441 5939 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:29:47.592637 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.592617 5939 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:29:47.592637 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.592635 5939 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:29:47.593187 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.593167 5939 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:29:47.596228 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596206 5939 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:29:47.596228 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596225 5939 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:29:47.596355 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596340 5939 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 00:29:47.596539 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596528 5939 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:29:47.596570 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596557 5939 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:29:47.596689 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596679 5939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:29:47.596731 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596697 5939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:29:47.596731 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596706 5939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:29:47.596789 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596737 5939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:29:47.596789 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596744 5939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:29:47.596789 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596749 5939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:29:47.596789 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596755 5939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:29:47.596789 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596761 5939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:29:47.596789 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596768 5939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:29:47.596789 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596774 5939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:29:47.596789 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596782 5939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:29:47.596789 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596791 5939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:29:47.597097 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596817 5939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:29:47.597097 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.596823 5939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:29:47.600327 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.600310 5939 server.go:1263] "Started kubelet" Oct 31 00:29:47.600464 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.600393 5939 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:29:47.601052 ip-10-0-143-213 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:29:47.601857 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.601535 5939 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:29:47.601857 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.601636 5939 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:29:47.602622 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.602594 5939 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:29:47.607336 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.603451 5939 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:29:47.608915 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.608899 5939 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:29:47.608998 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.608929 5939 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:29:47.608998 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:47.608970 5939 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-213.us-east-2.compute.internal\" not found" Oct 31 00:29:47.609096 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.609000 5939 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:29:47.609096 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.609011 5939 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:29:47.609096 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.609050 5939 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:23:37 +0000 UTC" deadline="2028-06-29 14:29:03.368213801 +0000 UTC" Oct 31 00:29:47.609096 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.609069 5939 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14581h59m15.759147395s" Oct 31 00:29:47.609096 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.609085 5939 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:29:47.611529 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.611509 5939 factory.go:55] Registering systemd factory Oct 31 00:29:47.611529 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.611531 5939 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:29:47.611840 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.611820 5939 factory.go:153] Registering CRI-O factory Oct 31 00:29:47.611906 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.611844 5939 factory.go:223] Registration of the crio container factory successfully Oct 31 00:29:47.611906 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.611899 5939 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:29:47.612024 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.611917 5939 factory.go:103] Registering Raw factory Oct 31 00:29:47.612024 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.611932 5939 manager.go:1199] Started watching for new ooms in manager Oct 31 00:29:47.612227 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.612205 5939 manager.go:319] Starting recovery of all containers Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.614653 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0" volumeName="kubernetes.io/projected/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-kube-api-access-q8dl6" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.614710 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.614729 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57decfb3-58bb-4b9c-b834-205594071bdf" volumeName="kubernetes.io/projected/57decfb3-58bb-4b9c-b834-205594071bdf-kube-api-access-v6j5l" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.614743 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e13c77-adcc-4f59-b1a4-93d090e2379e" volumeName="kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.614787 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d06594d-071d-47bb-ade5-16298fd91e2c" volumeName="kubernetes.io/secret/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-tls" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.614799 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79b89acd-a759-4036-bd91-16ee9ad9e288" volumeName="kubernetes.io/configmap/79b89acd-a759-4036-bd91-16ee9ad9e288-serviceca" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.614814 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.614827 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-config" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.614842 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40c70642-d50d-4f92-8ca1-10015c1a40cd" volumeName="kubernetes.io/configmap/40c70642-d50d-4f92-8ca1-10015c1a40cd-metrics-client-ca" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.614864 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57decfb3-58bb-4b9c-b834-205594071bdf" volumeName="kubernetes.io/secret/57decfb3-58bb-4b9c-b834-205594071bdf-cert" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.614878 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e13c77-adcc-4f59-b1a4-93d090e2379e" volumeName="kubernetes.io/projected/86e13c77-adcc-4f59-b1a4-93d090e2379e-kube-api-access-bmzx8" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.614888 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e13c77-adcc-4f59-b1a4-93d090e2379e" volumeName="kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.614914 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91208431-95bb-4e24-ab4b-e0f80f0d6207" volumeName="kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.614935 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba178af9-0f61-4c7b-a5ab-36e772495016" volumeName="kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.614952 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-config-out" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.614962 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-web-config" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.614971 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e13c77-adcc-4f59-b1a4-93d090e2379e" volumeName="kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615005 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3073e91a-8363-4d2a-8774-635302d4b040" volumeName="kubernetes.io/configmap/3073e91a-8363-4d2a-8774-635302d4b040-cni-binary-copy" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615018 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a8478f3-fe5c-4c04-b0a7-08bb736f3ede" volumeName="kubernetes.io/empty-dir/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-tmp" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615027 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d06594d-071d-47bb-ade5-16298fd91e2c" volumeName="kubernetes.io/configmap/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615034 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37dd17e5-ba62-4f95-bfc7-05b132104e44" volumeName="kubernetes.io/empty-dir/37dd17e5-ba62-4f95-bfc7-05b132104e44-tmp-dir" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615042 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6dca6aeb-205b-4634-9216-a98ba3957025" volumeName="kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615061 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40c70642-d50d-4f92-8ca1-10015c1a40cd" volumeName="kubernetes.io/secret/40c70642-d50d-4f92-8ca1-10015c1a40cd-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615069 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2eb2a42-2573-414d-8516-aac46f430550" volumeName="kubernetes.io/projected/b2eb2a42-2573-414d-8516-aac46f430550-kube-api-access-xwjlt" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615080 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfa459c8-39bb-4020-a514-0e03536a3dda" volumeName="kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615102 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="266a13d4-3f45-44d6-ac0e-3e171ce4e928" volumeName="kubernetes.io/secret/266a13d4-3f45-44d6-ac0e-3e171ce4e928-monitoring-plugin-cert" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615112 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f94485d7-b1a8-4caa-bfac-39c7cf29bfaa" volumeName="kubernetes.io/empty-dir/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-tmp-dir" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615119 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40c70642-d50d-4f92-8ca1-10015c1a40cd" volumeName="kubernetes.io/projected/40c70642-d50d-4f92-8ca1-10015c1a40cd-kube-api-access-qcx7m" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615138 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51dbc54b-218b-4274-9ebb-4cdbab400d22" volumeName="kubernetes.io/empty-dir/51dbc54b-218b-4274-9ebb-4cdbab400d22-data-volume" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615182 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfa459c8-39bb-4020-a514-0e03536a3dda" volumeName="kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-main-db" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615191 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-metrics-client-ca" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615199 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="266a13d4-3f45-44d6-ac0e-3e171ce4e928" volumeName="kubernetes.io/projected/266a13d4-3f45-44d6-ac0e-3e171ce4e928-kube-api-access-29dqz" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615206 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e13c77-adcc-4f59-b1a4-93d090e2379e" volumeName="kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615223 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91208431-95bb-4e24-ab4b-e0f80f0d6207" volumeName="kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615240 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfa459c8-39bb-4020-a514-0e03536a3dda" volumeName="kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-cluster-tls-config" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615248 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-metrics-client-certs" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615255 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615264 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1db4e02b-1e7a-4361-994e-f30507dbf0e7" volumeName="kubernetes.io/projected/1db4e02b-1e7a-4361-994e-f30507dbf0e7-kube-api-access-gcmn7" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615271 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f94485d7-b1a8-4caa-bfac-39c7cf29bfaa" volumeName="kubernetes.io/projected/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-kube-api-access-lr2lm" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615280 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3073e91a-8363-4d2a-8774-635302d4b040" volumeName="kubernetes.io/configmap/3073e91a-8363-4d2a-8774-635302d4b040-multus-daemon-config" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615295 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4677743-13cb-4a3a-b1df-8bd6baef94f4" volumeName="kubernetes.io/configmap/a4677743-13cb-4a3a-b1df-8bd6baef94f4-metrics-client-ca" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615302 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4677743-13cb-4a3a-b1df-8bd6baef94f4" volumeName="kubernetes.io/secret/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615318 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfa459c8-39bb-4020-a514-0e03536a3dda" volumeName="kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-config-out" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615327 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfa459c8-39bb-4020-a514-0e03536a3dda" volumeName="kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-metrics-client-ca" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615334 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7e79efa-c2ed-4cc0-928c-e0b9abf5275b" volumeName="kubernetes.io/projected/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-kube-api-access-xh2g8" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615341 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-db" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615349 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51dbc54b-218b-4274-9ebb-4cdbab400d22" volumeName="kubernetes.io/secret/51dbc54b-218b-4274-9ebb-4cdbab400d22-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615356 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d06594d-071d-47bb-ade5-16298fd91e2c" volumeName="kubernetes.io/empty-dir/8d06594d-071d-47bb-ade5-16298fd91e2c-volume-directive-shadow" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615363 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d06594d-071d-47bb-ade5-16298fd91e2c" volumeName="kubernetes.io/projected/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-api-access-bthrt" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615371 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91208431-95bb-4e24-ab4b-e0f80f0d6207" volumeName="kubernetes.io/empty-dir/91208431-95bb-4e24-ab4b-e0f80f0d6207-tmp" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615378 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91208431-95bb-4e24-ab4b-e0f80f0d6207" volumeName="kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615385 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615393 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6dca6aeb-205b-4634-9216-a98ba3957025" volumeName="kubernetes.io/projected/6dca6aeb-205b-4634-9216-a98ba3957025-kube-api-access-26png" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615402 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2eb2a42-2573-414d-8516-aac46f430550" volumeName="kubernetes.io/configmap/b2eb2a42-2573-414d-8516-aac46f430550-metrics-server-audit-profiles" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615410 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e13c77-adcc-4f59-b1a4-93d090e2379e" volumeName="kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-tls" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615417 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2eb2a42-2573-414d-8516-aac46f430550" volumeName="kubernetes.io/empty-dir/b2eb2a42-2573-414d-8516-aac46f430550-audit-log" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615435 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba178af9-0f61-4c7b-a5ab-36e772495016" volumeName="kubernetes.io/projected/ba178af9-0f61-4c7b-a5ab-36e772495016-kube-api-access-b8mpp" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615449 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfa459c8-39bb-4020-a514-0e03536a3dda" volumeName="kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615460 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c892baf2-ce51-4c47-8d79-f75f4f50c09d" volumeName="kubernetes.io/configmap/c892baf2-ce51-4c47-8d79-f75f4f50c09d-konnectivity-ca" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615469 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0" volumeName="kubernetes.io/configmap/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-iptables-alerter-script" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615476 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3073e91a-8363-4d2a-8774-635302d4b040" volumeName="kubernetes.io/projected/3073e91a-8363-4d2a-8774-635302d4b040-kube-api-access-4wvxm" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615484 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e13c77-adcc-4f59-b1a4-93d090e2379e" volumeName="kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-grpc-tls" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615493 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2eb2a42-2573-414d-8516-aac46f430550" volumeName="kubernetes.io/configmap/b2eb2a42-2573-414d-8516-aac46f430550-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615500 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7e79efa-c2ed-4cc0-928c-e0b9abf5275b" volumeName="kubernetes.io/secret/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615507 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37dd17e5-ba62-4f95-bfc7-05b132104e44" volumeName="kubernetes.io/secret/37dd17e5-ba62-4f95-bfc7-05b132104e44-metrics-tls" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615515 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6dca6aeb-205b-4634-9216-a98ba3957025" volumeName="kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-cni-binary-copy" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615522 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d06594d-071d-47bb-ade5-16298fd91e2c" volumeName="kubernetes.io/configmap/8d06594d-071d-47bb-ade5-16298fd91e2c-metrics-client-ca" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615529 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfa459c8-39bb-4020-a514-0e03536a3dda" volumeName="kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-kube-api-access-sc9ss" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615538 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51dbc54b-218b-4274-9ebb-4cdbab400d22" volumeName="kubernetes.io/projected/51dbc54b-218b-4274-9ebb-4cdbab400d22-kube-api-access-h6gnm" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615545 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2eb2a42-2573-414d-8516-aac46f430550" volumeName="kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-client-ca-bundle" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615555 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfa459c8-39bb-4020-a514-0e03536a3dda" volumeName="kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-tls-assets" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615562 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfa459c8-39bb-4020-a514-0e03536a3dda" volumeName="kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-web-config" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615571 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-grpc-tls" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615581 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a8478f3-fe5c-4c04-b0a7-08bb736f3ede" volumeName="kubernetes.io/projected/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-kube-api-access-j8cjz" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615588 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f86ab44-a957-4e07-8bdf-5a8f813f27d4" volumeName="kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615595 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4677743-13cb-4a3a-b1df-8bd6baef94f4" volumeName="kubernetes.io/empty-dir/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-textfile" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615603 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4677743-13cb-4a3a-b1df-8bd6baef94f4" volumeName="kubernetes.io/secret/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-tls" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615610 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615617 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40c70642-d50d-4f92-8ca1-10015c1a40cd" volumeName="kubernetes.io/secret/40c70642-d50d-4f92-8ca1-10015c1a40cd-openshift-state-metrics-tls" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615635 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e13c77-adcc-4f59-b1a4-93d090e2379e" volumeName="kubernetes.io/configmap/86e13c77-adcc-4f59-b1a4-93d090e2379e-metrics-client-ca" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615642 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-kube-api-access-tbxs7" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615649 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91208431-95bb-4e24-ab4b-e0f80f0d6207" volumeName="kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615657 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37dd17e5-ba62-4f95-bfc7-05b132104e44" volumeName="kubernetes.io/configmap/37dd17e5-ba62-4f95-bfc7-05b132104e44-config-volume" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615696 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e13c77-adcc-4f59-b1a4-93d090e2379e" volumeName="kubernetes.io/empty-dir/86e13c77-adcc-4f59-b1a4-93d090e2379e-tmp" seLinuxMountContext="" Oct 31 00:29:47.615734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615707 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfa459c8-39bb-4020-a514-0e03536a3dda" volumeName="kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615962 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7b81351-0c71-49b6-bd86-e48483b8b555" volumeName="kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615982 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.615995 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4677743-13cb-4a3a-b1df-8bd6baef94f4" volumeName="kubernetes.io/configmap/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616018 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51dbc54b-218b-4274-9ebb-4cdbab400d22" volumeName="kubernetes.io/configmap/51dbc54b-218b-4274-9ebb-4cdbab400d22-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616031 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79b89acd-a759-4036-bd91-16ee9ad9e288" volumeName="kubernetes.io/projected/79b89acd-a759-4036-bd91-16ee9ad9e288-kube-api-access-mdn2m" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616049 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfa459c8-39bb-4020-a514-0e03536a3dda" volumeName="kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616061 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616074 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a8478f3-fe5c-4c04-b0a7-08bb736f3ede" volumeName="kubernetes.io/empty-dir/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-tuned" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616090 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2eb2a42-2573-414d-8516-aac46f430550" volumeName="kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-secret-metrics-server-tls" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616102 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91208431-95bb-4e24-ab4b-e0f80f0d6207" volumeName="kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616142 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2eb2a42-2573-414d-8516-aac46f430550" volumeName="kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616155 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfa459c8-39bb-4020-a514-0e03536a3dda" volumeName="kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616171 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c892baf2-ce51-4c47-8d79-f75f4f50c09d" volumeName="kubernetes.io/secret/c892baf2-ce51-4c47-8d79-f75f4f50c09d-agent-certs" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616189 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7e79efa-c2ed-4cc0-928c-e0b9abf5275b" volumeName="kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovnkube-config" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616201 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616227 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616239 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6dca6aeb-205b-4634-9216-a98ba3957025" volumeName="kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616285 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91208431-95bb-4e24-ab4b-e0f80f0d6207" volumeName="kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616303 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91208431-95bb-4e24-ab4b-e0f80f0d6207" volumeName="kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616315 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616334 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-tls-assets" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616346 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d06594d-071d-47bb-ade5-16298fd91e2c" volumeName="kubernetes.io/secret/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616358 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37dd17e5-ba62-4f95-bfc7-05b132104e44" volumeName="kubernetes.io/projected/37dd17e5-ba62-4f95-bfc7-05b132104e44-kube-api-access-wvdnb" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616379 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4677743-13cb-4a3a-b1df-8bd6baef94f4" volumeName="kubernetes.io/projected/a4677743-13cb-4a3a-b1df-8bd6baef94f4-kube-api-access-22g5c" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616392 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfa459c8-39bb-4020-a514-0e03536a3dda" volumeName="kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-config-volume" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616410 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7e79efa-c2ed-4cc0-928c-e0b9abf5275b" volumeName="kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovnkube-script-lib" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616879 5939 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.616421 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7e79efa-c2ed-4cc0-928c-e0b9abf5275b" volumeName="kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-env-overrides" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.618053 5939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc16559-1d24-40ea-9ba6-aa6896178faf" volumeName="kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.618075 5939 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.618083 5939 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:29:47.619965 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.618921 5939 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:29:47.621762 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.621741 5939 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:29:47.623906 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.623886 5939 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 00:29:47.644819 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.644774 5939 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:29:47.646452 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.646423 5939 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:29:47.646452 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.646449 5939 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:29:47.646600 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.646469 5939 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:29:47.646600 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:47.646510 5939 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 00:29:47.654614 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.654575 5939 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:29:47.661598 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.661576 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:29:47.666200 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.666165 5939 generic.go:356] "Generic (PLEG): container finished" podID="a4677743-13cb-4a3a-b1df-8bd6baef94f4" containerID="135b24dfdec8cceba8ebbed7ecf5d5bca0766df94513156ae96bce986989b1b9" exitCode=0 Oct 31 00:29:47.667415 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.667390 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:29:47.679835 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.679809 5939 generic.go:356] "Generic (PLEG): container finished" podID="bb8ff3cdd197b5db95c89f56a4f8fb1a" containerID="c7e41d6549c55b6bab121a73ba88850f604a59551b1c761ab8ba2b33b3c214a2" exitCode=0 Oct 31 00:29:47.688745 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.688724 5939 generic.go:356] "Generic (PLEG): container finished" podID="6dca6aeb-205b-4634-9216-a98ba3957025" containerID="3c2ab6f88e9e540056a0ab52280e2b1c8760c41f2669da8a50f00c35c36a6431" exitCode=0 Oct 31 00:29:47.688745 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.688743 5939 generic.go:356] "Generic (PLEG): container finished" podID="6dca6aeb-205b-4634-9216-a98ba3957025" containerID="d61d0a845291a0f2cf37c076996db842683dc842f4a5f12f18a247a6bb8e24b0" exitCode=0 Oct 31 00:29:47.688879 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.688752 5939 generic.go:356] "Generic (PLEG): container finished" podID="6dca6aeb-205b-4634-9216-a98ba3957025" containerID="59a8d5ac93a28fada17d7d14a8c8b947ebafa5602198f214f31813b8170d4665" exitCode=0 Oct 31 00:29:47.688879 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.688761 5939 generic.go:356] "Generic (PLEG): container finished" podID="6dca6aeb-205b-4634-9216-a98ba3957025" containerID="3af11775f85dfdcaaeae708b1732561ab92b7ddcac63ddcd905d3c5d2b274177" exitCode=0 Oct 31 00:29:47.688879 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.688771 5939 generic.go:356] "Generic (PLEG): container finished" podID="6dca6aeb-205b-4634-9216-a98ba3957025" containerID="a98f081c5e926131f2498738aaea20a4b13790a56f07a2229b89a9b1a7f767ce" exitCode=0 Oct 31 00:29:47.688879 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.688779 5939 generic.go:356] "Generic (PLEG): container finished" podID="6dca6aeb-205b-4634-9216-a98ba3957025" containerID="8ce55a93270d2f9f8f1ad8bc617a29813da3fc0a132c2a9a9d693cb9012f57eb" exitCode=0 Oct 31 00:29:47.694458 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.694428 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:29:47.694868 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.694844 5939 generic.go:356] "Generic (PLEG): container finished" podID="e7e79efa-c2ed-4cc0-928c-e0b9abf5275b" containerID="ae6d5c1751eeff5b784efbc14231c8b255feba4c2ed10b76dedb026486e5f0a3" exitCode=1 Oct 31 00:29:47.699129 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.699094 5939 generic.go:356] "Generic (PLEG): container finished" podID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerID="eb74ac6c958294b3fdf7f5e08d3c3921e026e9c566c85820664a00f8705ccb49" exitCode=0 Oct 31 00:29:47.704597 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.704573 5939 generic.go:356] "Generic (PLEG): container finished" podID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerID="fb92b869942ab49a2d6edb574af10bc1fcb0d928852268c7469de8726e518eef" exitCode=0 Oct 31 00:29:47.744179 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.744124 5939 manager.go:324] Recovery completed Oct 31 00:29:47.746578 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:47.746557 5939 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 00:29:47.769293 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.769269 5939 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:29:47.769293 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.769284 5939 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:29:47.769442 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.769301 5939 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:29:47.769442 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.769427 5939 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 00:29:47.769619 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.769438 5939 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 00:29:47.769619 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.769454 5939 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 00:29:47.769619 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.769459 5939 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 00:29:47.769619 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.769464 5939 policy_none.go:50] "Start" Oct 31 00:29:47.769619 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.769473 5939 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:29:47.769619 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.769482 5939 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:29:47.770943 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.770927 5939 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 00:29:47.770943 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.770941 5939 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 00:29:47.771054 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.770947 5939 policy_none.go:44] "Start" Oct 31 00:29:47.775583 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.775552 5939 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:29:47.778736 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.778721 5939 manager.go:341] "Starting Device Plugin manager" Oct 31 00:29:47.779265 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:47.778745 5939 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:29:47.779265 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.779266 5939 server.go:87] "Starting device plugin registration server" Oct 31 00:29:47.779534 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.779522 5939 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:29:47.779586 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.779535 5939 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:29:47.779658 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.779637 5939 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:29:47.779854 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.779805 5939 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:29:47.779928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.779851 5939 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:29:47.779928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.779879 5939 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:29:47.780018 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.779942 5939 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:29:47.779839426Z","UUID":"87fcae28-d24e-44cb-a842-49157d336b04","Handler":null,"Name":"","Endpoint":""} Oct 31 00:29:47.781486 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.781472 5939 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:29:47.781596 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.781494 5939 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:29:47.782116 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:47.781895 5939 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:29:47.880341 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.880307 5939 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:29:47.883836 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.883813 5939 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:29:47.883953 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.883846 5939 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:29:47.883953 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.883856 5939 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-213.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:29:47.883953 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.883903 5939 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:47.902486 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.902455 5939 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:47.902613 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.902513 5939 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:47.947449 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.947381 5939 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal"] Oct 31 00:29:47.948102 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.948039 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" event={"ID":"bb8ff3cdd197b5db95c89f56a4f8fb1a","Type":"ContainerStarted","Data":"5f66a2620f261d20d7f3221521773a3b9b12def7aea77138a52b51d04540a24d"} Oct 31 00:29:47.948318 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.948112 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" event={"ID":"bb8ff3cdd197b5db95c89f56a4f8fb1a","Type":"ContainerDied","Data":"c7e41d6549c55b6bab121a73ba88850f604a59551b1c761ab8ba2b33b3c214a2"} Oct 31 00:29:47.948318 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.948135 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" event={"ID":"bb8ff3cdd197b5db95c89f56a4f8fb1a","Type":"ContainerStarted","Data":"34cac5d6a5a3bffaec384bbcac6431fd3ab274264163a27a4188dd9723401e3a"} Oct 31 00:29:47.948318 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.948084 5939 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:47.948318 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.948151 5939 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:47.948318 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.948281 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" event={"ID":"2b347d22e86cd58c1b6312f0f07ef259","Type":"ContainerStarted","Data":"15fdbf75cb7433c19267532a7075108f0d65abefd59b5a24912500e29de12c2b"} Oct 31 00:29:47.948318 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.948298 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" event={"ID":"2b347d22e86cd58c1b6312f0f07ef259","Type":"ContainerStarted","Data":"e768af5000c0dcee47d0b3b7dafcfd78487ca2a82c5a03555e371bb361caa13d"} Oct 31 00:29:47.970205 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.970173 5939 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:29:47.970312 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:47.970223 5939 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:47.971964 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:47.971953 5939 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:29:47.972002 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:47.971982 5939 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:48.026341 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.026279 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb8ff3cdd197b5db95c89f56a4f8fb1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"bb8ff3cdd197b5db95c89f56a4f8fb1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:48.026341 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.026311 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb8ff3cdd197b5db95c89f56a4f8fb1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"bb8ff3cdd197b5db95c89f56a4f8fb1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:48.026341 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.026331 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b347d22e86cd58c1b6312f0f07ef259-config\") pod \"kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"2b347d22e86cd58c1b6312f0f07ef259\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:48.126866 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.126827 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b347d22e86cd58c1b6312f0f07ef259-config\") pod \"kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"2b347d22e86cd58c1b6312f0f07ef259\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:48.126866 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.126870 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb8ff3cdd197b5db95c89f56a4f8fb1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"bb8ff3cdd197b5db95c89f56a4f8fb1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:48.127022 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.126887 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb8ff3cdd197b5db95c89f56a4f8fb1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"bb8ff3cdd197b5db95c89f56a4f8fb1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:48.127022 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.126937 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b347d22e86cd58c1b6312f0f07ef259-config\") pod \"kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"2b347d22e86cd58c1b6312f0f07ef259\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:48.127022 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.126937 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb8ff3cdd197b5db95c89f56a4f8fb1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"bb8ff3cdd197b5db95c89f56a4f8fb1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:48.127022 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.126938 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb8ff3cdd197b5db95c89f56a4f8fb1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal\" (UID: \"bb8ff3cdd197b5db95c89f56a4f8fb1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:48.592545 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.592512 5939 apiserver.go:52] "Watching apiserver" Oct 31 00:29:48.606162 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.606142 5939 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:29:48.606827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.606805 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-dhwx4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst","openshift-image-registry/node-ca-9l52l","openshift-monitoring/metrics-server-67cc7bf54c-zxwf4","openshift-monitoring/node-exporter-vl8l6","openshift-cluster-node-tuning-operator/tuned-dwxbv","openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2","kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal","openshift-console/console-74f876fbf6-4mlmt","kube-system/konnectivity-agent-b64jh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal","openshift-monitoring/alertmanager-main-0","openshift-multus/multus-pghrd","openshift-multus/network-metrics-daemon-jcdhx","openshift-network-diagnostics/network-check-target-zwhbr","openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk","openshift-multus/multus-additional-cni-plugins-dcljc","openshift-dns/dns-default-xkqrm","openshift-dns/node-resolver-wvff6","openshift-ingress-canary/ingress-canary-pnmtp","openshift-insights/insights-runtime-extractor-w49pc","openshift-network-operator/iptables-alerter-sm6rp","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/monitoring-plugin-77f4f76594-65wl9","openshift-monitoring/thanos-querier-7bcdf957ff-s472c","kube-system/global-pull-secret-syncer-dvl2d"] Oct 31 00:29:48.614056 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.614027 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:29:48.614228 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.614208 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:29:48.614586 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.614564 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:29:48.614889 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.614872 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:29:48.615042 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.615020 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:29:48.615131 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.615109 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:29:48.615304 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.615286 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:29:48.615382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.615369 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:29:48.615471 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.615455 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:29:48.615633 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.615600 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:29:48.615726 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.615636 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:29:48.615803 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.615778 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:29:48.615920 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.615907 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:29:48.616000 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.615982 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:29:48.616320 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.616301 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qx27s\"" Oct 31 00:29:48.616320 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.616315 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:29:48.616790 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.615951 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:29:48.616896 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.616878 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:29:48.616947 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.616894 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:29:48.617048 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.617027 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:29:48.617126 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.616776 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b2pkn\"" Oct 31 00:29:48.617250 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.616776 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hxhg5\"" Oct 31 00:29:48.617250 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.617173 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:29:48.617250 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.617229 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-56d49\"" Oct 31 00:29:48.617846 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.617825 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:29:48.617925 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.617863 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:29:48.617925 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.617893 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j8g45\"" Oct 31 00:29:48.617994 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.617949 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4bqfb\"" Oct 31 00:29:48.617994 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.617962 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dnwdb\"" Oct 31 00:29:48.618078 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.618003 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:29:48.618078 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.618051 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-g4hxn\"" Oct 31 00:29:48.618161 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.618146 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:29:48.618260 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.618244 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:29:48.618349 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.618335 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9bm5h\"" Oct 31 00:29:48.618581 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.618565 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:29:48.618702 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.618608 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cnz4r\"" Oct 31 00:29:48.618702 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.618698 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:29:48.618851 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.618713 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:29:48.618851 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.618715 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:29:48.618851 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.618743 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:29:48.618851 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.618777 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:29:48.618851 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.618844 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r62gb\"" Oct 31 00:29:48.619093 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.618926 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wzwtm\"" Oct 31 00:29:48.619396 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.619384 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:29:48.619475 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.619445 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8j7n8\"" Oct 31 00:29:48.619475 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.619453 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:29:48.622833 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.622807 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:29:48.623047 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.623031 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:29:48.623133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.623117 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:29:48.623347 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.623331 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nchkl\"" Oct 31 00:29:48.623410 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.623397 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:29:48.623469 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.623427 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:29:48.623523 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.623490 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wfrwt\"" Oct 31 00:29:48.623569 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.623518 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:29:48.623691 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.623656 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:29:48.626225 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.626204 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:29:48.626506 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.626492 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:29:48.627032 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.627019 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:29:48.627318 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.627307 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 00:29:48.627475 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.627461 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:29:48.628597 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.628580 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysctl-d\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.628657 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.628608 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8mpp\" (UniqueName: \"kubernetes.io/projected/ba178af9-0f61-4c7b-a5ab-36e772495016-kube-api-access-b8mpp\") pod \"network-metrics-daemon-jcdhx\" (UID: \"ba178af9-0f61-4c7b-a5ab-36e772495016\") " pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:29:48.628657 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.628627 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-systemd-units\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.628657 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.628642 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-run-netns\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.628797 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.628694 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-etc-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.628797 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.628716 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-ovn\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.628797 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.628758 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-sys\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.628797 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.628777 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovnkube-config\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.628950 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.628837 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovnkube-script-lib\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.628950 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.628871 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:48.629055 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.628956 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysconfig\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.629055 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629006 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-systemd\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.629055 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629046 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-lib-modules\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.629195 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629069 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-slash\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.629195 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629069 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovnkube-config\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.629195 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629125 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2lm\" (UniqueName: \"kubernetes.io/projected/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-kube-api-access-lr2lm\") pod \"node-resolver-wvff6\" (UID: \"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa\") " pod="openshift-dns/node-resolver-wvff6" Oct 31 00:29:48.629195 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629156 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7b81351-0c71-49b6-bd86-e48483b8b555-dbus\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:48.629195 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629182 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-kubernetes\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.629195 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629175 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovnkube-script-lib\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.629521 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629238 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs\") pod \"network-metrics-daemon-jcdhx\" (UID: \"ba178af9-0f61-4c7b-a5ab-36e772495016\") " pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:29:48.629521 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629260 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 00:29:48.629521 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629255 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7b81351-0c71-49b6-bd86-e48483b8b555-original-pull-secret\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:48.629521 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629378 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.629521 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629404 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.629521 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629438 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-env-overrides\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.629521 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629479 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-hosts-file\") pod \"node-resolver-wvff6\" (UID: \"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa\") " pod="openshift-dns/node-resolver-wvff6" Oct 31 00:29:48.629521 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629506 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-tmp-dir\") pod \"node-resolver-wvff6\" (UID: \"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa\") " pod="openshift-dns/node-resolver-wvff6" Oct 31 00:29:48.629834 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629568 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba178af9-0f61-4c7b-a5ab-36e772495016-metrics-certs\") pod \"network-metrics-daemon-jcdhx\" (UID: \"ba178af9-0f61-4c7b-a5ab-36e772495016\") " pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:29:48.629834 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629574 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-tmp-dir\") pod \"node-resolver-wvff6\" (UID: \"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa\") " pod="openshift-dns/node-resolver-wvff6" Oct 31 00:29:48.629834 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629607 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-env-overrides\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.629834 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629563 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7b81351-0c71-49b6-bd86-e48483b8b555-kubelet-config\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:48.629834 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629647 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-modprobe-d\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.629834 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629713 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysctl-conf\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.629834 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629743 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-var-lib-kubelet\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.629834 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629762 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-host\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.629834 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629794 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-systemd\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.629834 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629820 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-var-lib-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.630112 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629858 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-node-log\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.630112 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629886 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-log-socket\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.630112 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629901 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-run\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.630112 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.629941 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-tuned\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.630112 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.630017 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-tmp\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.630112 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.630058 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-tuned\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.630112 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.630079 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-tmp\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.630307 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.630139 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-kubelet\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.630307 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.630172 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-cni-netd\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.630307 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.630205 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.630307 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.630252 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovn-node-metrics-cert\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.630307 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.630281 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh2g8\" (UniqueName: \"kubernetes.io/projected/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-kube-api-access-xh2g8\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.630437 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.630306 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8cjz\" (UniqueName: \"kubernetes.io/projected/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-kube-api-access-j8cjz\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.630437 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.630380 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-cni-bin\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.630437 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.630425 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lkc\" (UniqueName: \"kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc\") pod \"network-check-target-zwhbr\" (UID: \"6f86ab44-a957-4e07-8bdf-5a8f813f27d4\") " pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:48.630651 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.630636 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-ovn-node-metrics-cert\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.631255 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.631239 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:29:48.631339 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.631322 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kr8zh\"" Oct 31 00:29:48.631369 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.631322 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:29:48.631369 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.631347 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:29:48.631433 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.631375 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wf9pg\"" Oct 31 00:29:48.631461 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.631430 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 00:29:48.641240 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.641223 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8mpp\" (UniqueName: \"kubernetes.io/projected/ba178af9-0f61-4c7b-a5ab-36e772495016-kube-api-access-b8mpp\") pod \"network-metrics-daemon-jcdhx\" (UID: \"ba178af9-0f61-4c7b-a5ab-36e772495016\") " pod="openshift-multus/network-metrics-daemon-jcdhx" Oct 31 00:29:48.642956 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.642941 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8cjz\" (UniqueName: \"kubernetes.io/projected/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-kube-api-access-j8cjz\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.643952 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.643933 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2lm\" (UniqueName: \"kubernetes.io/projected/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-kube-api-access-lr2lm\") pod \"node-resolver-wvff6\" (UID: \"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa\") " pod="openshift-dns/node-resolver-wvff6" Oct 31 00:29:48.644349 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.644334 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7lkc\" (UniqueName: \"kubernetes.io/projected/6f86ab44-a957-4e07-8bdf-5a8f813f27d4-kube-api-access-f7lkc\") pod \"network-check-target-zwhbr\" (UID: \"6f86ab44-a957-4e07-8bdf-5a8f813f27d4\") " pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:48.644457 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.644442 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh2g8\" (UniqueName: \"kubernetes.io/projected/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-kube-api-access-xh2g8\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.711141 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.711121 5939 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:29:48.714690 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.714655 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 00:29:48.714931 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.714910 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:29:48.715095 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.714798 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:29:48.715246 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.714694 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:29:48.715476 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.715455 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9gnrx\"" Oct 31 00:29:48.715821 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.715802 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 00:29:48.715998 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.715977 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:29:48.716101 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.716083 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 00:29:48.716320 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.716302 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 00:29:48.716393 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.716344 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 00:29:48.716602 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.716588 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 00:29:48.716705 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.715807 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:29:48.716793 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.716083 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:29:48.716885 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.716866 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 00:29:48.717069 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.717043 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:29:48.717130 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.717103 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:29:48.717180 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.717132 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:29:48.717329 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.717272 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:29:48.717543 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.717275 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:29:48.717862 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.717827 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bzqj7\"" Oct 31 00:29:48.717960 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.717881 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:29:48.718008 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.717997 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:29:48.718085 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.718069 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5q9q9rm8kcsp5\"" Oct 31 00:29:48.718183 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.718117 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2imsi3d4ipqml\"" Oct 31 00:29:48.718241 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.718229 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:29:48.718327 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.718311 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pdw62\"" Oct 31 00:29:48.718397 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.718333 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-84jxj\"" Oct 31 00:29:48.718397 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.718379 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1pefrp1lki40c\"" Oct 31 00:29:48.718656 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.718479 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:29:48.718656 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.717834 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:29:48.718656 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.718544 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 00:29:48.718656 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.718598 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:29:48.718656 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.718481 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lwv25\"" Oct 31 00:29:48.718656 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.718640 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 00:29:48.718918 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.718707 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:29:48.718918 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.718810 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-n2kx5\"" Oct 31 00:29:48.718918 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.718874 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:29:48.718918 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.718881 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:29:48.720178 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.720158 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:29:48.720659 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.720641 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 00:29:48.720727 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.720711 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:29:48.721066 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.721052 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:29:48.721385 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.721368 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:29:48.730904 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.730884 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.731013 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.730913 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79b89acd-a759-4036-bd91-16ee9ad9e288-serviceca\") pod \"node-ca-9l52l\" (UID: \"79b89acd-a759-4036-bd91-16ee9ad9e288\") " pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:29:48.731013 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.730929 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37dd17e5-ba62-4f95-bfc7-05b132104e44-metrics-tls\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:48.731013 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.730945 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37dd17e5-ba62-4f95-bfc7-05b132104e44-tmp-dir\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:48.731013 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.730965 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-sys-fs\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.731013 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.730984 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-run-netns\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.731013 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731001 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-etc-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.731299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731020 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8dl6\" (UniqueName: \"kubernetes.io/projected/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-kube-api-access-q8dl6\") pod \"iptables-alerter-sm6rp\" (UID: \"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0\") " pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:29:48.731299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731037 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-conf-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.731299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731058 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.731299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731057 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37dd17e5-ba62-4f95-bfc7-05b132104e44-tmp-dir\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:48.731299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731075 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.731299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731091 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcmn7\" (UniqueName: \"kubernetes.io/projected/1db4e02b-1e7a-4361-994e-f30507dbf0e7-kube-api-access-gcmn7\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.731299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731105 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-sys\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.731299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731125 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26png\" (UniqueName: \"kubernetes.io/projected/6dca6aeb-205b-4634-9216-a98ba3957025-kube-api-access-26png\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:29:48.731299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731141 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.731299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731167 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.731299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731184 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.731299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731205 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysconfig\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.731299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731204 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37dd17e5-ba62-4f95-bfc7-05b132104e44-metrics-tls\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:48.731299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731220 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-systemd\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.731299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731256 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:48.731299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731265 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-systemd\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.731299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731300 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3073e91a-8363-4d2a-8774-635302d4b040-cni-binary-copy\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731318 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731329 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731346 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731365 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.731369 5939 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731384 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731400 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/266a13d4-3f45-44d6-ac0e-3e171ce4e928-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-65wl9\" (UID: \"266a13d4-3f45-44d6-ac0e-3e171ce4e928\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731414 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6j5l\" (UniqueName: \"kubernetes.io/projected/57decfb3-58bb-4b9c-b834-205594071bdf-kube-api-access-v6j5l\") pod \"ingress-canary-pnmtp\" (UID: \"57decfb3-58bb-4b9c-b834-205594071bdf\") " pod="openshift-ingress-canary/ingress-canary-pnmtp" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.731443 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:49.231423532 +0000 UTC m=+1.720698705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731463 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3073e91a-8363-4d2a-8774-635302d4b040-multus-daemon-config\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731472 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3073e91a-8363-4d2a-8774-635302d4b040-cni-binary-copy\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731491 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731516 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51dbc54b-218b-4274-9ebb-4cdbab400d22-crio-socket\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731541 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6gnm\" (UniqueName: \"kubernetes.io/projected/51dbc54b-218b-4274-9ebb-4cdbab400d22-kube-api-access-h6gnm\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731563 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-host\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731585 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-systemd\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731608 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-node-log\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731631 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-hostroot\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731636 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79b89acd-a759-4036-bd91-16ee9ad9e288-serviceca\") pod \"node-ca-9l52l\" (UID: \"79b89acd-a759-4036-bd91-16ee9ad9e288\") " pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731654 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731702 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51dbc54b-218b-4274-9ebb-4cdbab400d22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731728 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-run\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731773 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731787 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731859 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-node-log\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.731871 5939 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.731935 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:49.231917158 +0000 UTC m=+1.721192329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731952 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3073e91a-8363-4d2a-8774-635302d4b040-multus-daemon-config\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731984 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-host\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.731987 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51dbc54b-218b-4274-9ebb-4cdbab400d22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732056 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:48.732067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732036 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysconfig\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732092 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732119 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732132 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732143 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-config\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732171 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdn2m\" (UniqueName: \"kubernetes.io/projected/79b89acd-a759-4036-bd91-16ee9ad9e288-kube-api-access-mdn2m\") pod \"node-ca-9l52l\" (UID: \"79b89acd-a759-4036-bd91-16ee9ad9e288\") " pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732198 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-system-cni-dir\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732200 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/266a13d4-3f45-44d6-ac0e-3e171ce4e928-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-65wl9\" (UID: \"266a13d4-3f45-44d6-ac0e-3e171ce4e928\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732224 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732229 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732285 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-run-netns\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732320 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-etc-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732398 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732447 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732518 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732527 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732561 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-cnibin\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.732599 5939 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732635 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-run\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.732645 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:49.23263359 +0000 UTC m=+1.721908750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-config" not registered Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732691 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-systemd\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732728 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-sys\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732596 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732751 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732756 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-config-volume\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732849 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732887 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-config-volume\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732889 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732925 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732945 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79b89acd-a759-4036-bd91-16ee9ad9e288-host\") pod \"node-ca-9l52l\" (UID: \"79b89acd-a759-4036-bd91-16ee9ad9e288\") " pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732973 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40c70642-d50d-4f92-8ca1-10015c1a40cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.732991 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733002 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-config\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733007 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wvxm\" (UniqueName: \"kubernetes.io/projected/3073e91a-8363-4d2a-8774-635302d4b040-kube-api-access-4wvxm\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733160 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d06594d-071d-47bb-ade5-16298fd91e2c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733193 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7b81351-0c71-49b6-bd86-e48483b8b555-dbus\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733240 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733248 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29dqz\" (UniqueName: \"kubernetes.io/projected/266a13d4-3f45-44d6-ac0e-3e171ce4e928-kube-api-access-29dqz\") pod \"monitoring-plugin-77f4f76594-65wl9\" (UID: \"266a13d4-3f45-44d6-ac0e-3e171ce4e928\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733282 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-multus-certs\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733299 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733316 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733351 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-host-slash\") pod \"iptables-alerter-sm6rp\" (UID: \"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0\") " pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733366 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-k8s-cni-cncf-io\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.733364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733383 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-web-config\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733401 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8d06594d-071d-47bb-ade5-16298fd91e2c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733418 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2eb2a42-2573-414d-8516-aac46f430550-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733477 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwjlt\" (UniqueName: \"kubernetes.io/projected/b2eb2a42-2573-414d-8516-aac46f430550-kube-api-access-xwjlt\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733494 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-textfile\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733515 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-cni-bin\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733530 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733546 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbxs7\" (UniqueName: \"kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-kube-api-access-tbxs7\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733573 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-modprobe-d\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733587 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-var-lib-kubelet\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733601 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-log-socket\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733618 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733640 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733802 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40c70642-d50d-4f92-8ca1-10015c1a40cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733817 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22g5c\" (UniqueName: \"kubernetes.io/projected/a4677743-13cb-4a3a-b1df-8bd6baef94f4-kube-api-access-22g5c\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733841 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-textfile\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733851 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733881 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/37dd17e5-ba62-4f95-bfc7-05b132104e44-config-volume\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733906 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7b81351-0c71-49b6-bd86-e48483b8b555-dbus\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733909 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-kubelet\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733951 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-kubelet\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733950 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734012 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-etc-kubernetes\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734049 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734070 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734078 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734106 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86e13c77-adcc-4f59-b1a4-93d090e2379e-tmp\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734133 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.733854 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8d06594d-071d-47bb-ade5-16298fd91e2c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734190 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-grpc-tls\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734219 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-root\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734230 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d06594d-071d-47bb-ade5-16298fd91e2c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734248 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc9ss\" (UniqueName: \"kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-kube-api-access-sc9ss\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734279 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734311 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734343 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-ovn\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734364 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-modprobe-d\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734371 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4677743-13cb-4a3a-b1df-8bd6baef94f4-metrics-client-ca\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734403 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2eb2a42-2573-414d-8516-aac46f430550-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734419 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-system-cni-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734450 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-web-config\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734477 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734507 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51dbc54b-218b-4274-9ebb-4cdbab400d22-data-volume\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734590 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-slash\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734617 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57decfb3-58bb-4b9c-b834-205594071bdf-cert\") pod \"ingress-canary-pnmtp\" (UID: \"57decfb3-58bb-4b9c-b834-205594071bdf\") " pod="openshift-ingress-canary/ingress-canary-pnmtp" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734631 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4677743-13cb-4a3a-b1df-8bd6baef94f4-metrics-client-ca\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734644 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mfhm\" (UniqueName: \"kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734637 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734713 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-tls\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734720 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86e13c77-adcc-4f59-b1a4-93d090e2379e-tmp\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734744 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-kubernetes\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734835 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734861 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734865 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.734889 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-cni-binary-copy\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735056 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-web-config\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.735111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735117 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-slash\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735177 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51dbc54b-218b-4274-9ebb-4cdbab400d22-data-volume\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735180 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735225 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-var-lib-kubelet\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735258 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-log-socket\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735426 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-tls\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735444 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735446 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-web-config\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735586 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735623 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-wtmp\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735844 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-ovn\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735683 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735871 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735892 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-run-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735940 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-cni-binary-copy\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735957 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-kubernetes\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.735966 5939 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735632 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57decfb3-58bb-4b9c-b834-205594071bdf-cert\") pod \"ingress-canary-pnmtp\" (UID: \"57decfb3-58bb-4b9c-b834-205594071bdf\") " pod="openshift-ingress-canary/ingress-canary-pnmtp" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.735997 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736028 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c892baf2-ce51-4c47-8d79-f75f4f50c09d-konnectivity-ca\") pod \"konnectivity-agent-b64jh\" (UID: \"c892baf2-ce51-4c47-8d79-f75f4f50c09d\") " pod="kube-system/konnectivity-agent-b64jh" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736067 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-cnibin\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.736107 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:49.236091473 +0000 UTC m=+1.725366644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-serving-cert" not registered Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736146 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bthrt\" (UniqueName: \"kubernetes.io/projected/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-api-access-bthrt\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736181 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcx7m\" (UniqueName: \"kubernetes.io/projected/40c70642-d50d-4f92-8ca1-10015c1a40cd-kube-api-access-qcx7m\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736392 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-grpc-tls\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736446 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-tls\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736486 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736684 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-cni-netd\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736714 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c892baf2-ce51-4c47-8d79-f75f4f50c09d-konnectivity-ca\") pod \"konnectivity-agent-b64jh\" (UID: \"c892baf2-ce51-4c47-8d79-f75f4f50c09d\") " pod="kube-system/konnectivity-agent-b64jh" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736719 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-sys\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736799 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-cni-bin\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736833 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40c70642-d50d-4f92-8ca1-10015c1a40cd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736872 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86e13c77-adcc-4f59-b1a4-93d090e2379e-metrics-client-ca\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736944 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736952 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmzx8\" (UniqueName: \"kubernetes.io/projected/86e13c77-adcc-4f59-b1a4-93d090e2379e-kube-api-access-bmzx8\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736978 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736988 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-cni-netd\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.736982 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737028 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-cni-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737058 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-cni-multus\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737089 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysctl-d\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737130 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-systemd-units\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737152 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737162 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737191 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-client-ca-bundle\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737199 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-cni-bin\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737219 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737246 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-socket-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737308 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-secret-metrics-server-client-certs\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737336 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b2eb2a42-2573-414d-8516-aac46f430550-metrics-server-audit-profiles\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737364 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737368 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40c70642-d50d-4f92-8ca1-10015c1a40cd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737390 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvdnb\" (UniqueName: \"kubernetes.io/projected/37dd17e5-ba62-4f95-bfc7-05b132104e44-kube-api-access-wvdnb\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737417 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51dbc54b-218b-4274-9ebb-4cdbab400d22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737442 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-lib-modules\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737480 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-iptables-alerter-script\") pod \"iptables-alerter-sm6rp\" (UID: \"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0\") " pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737510 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737538 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-device-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737566 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.737564 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737569 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/37dd17e5-ba62-4f95-bfc7-05b132104e44-config-volume\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737641 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-client-ca-bundle\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737643 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-netns\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737707 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737739 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-registration-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737769 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b2eb2a42-2573-414d-8516-aac46f430550-audit-log\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737798 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c892baf2-ce51-4c47-8d79-f75f4f50c09d-agent-certs\") pod \"konnectivity-agent-b64jh\" (UID: \"c892baf2-ce51-4c47-8d79-f75f4f50c09d\") " pod="kube-system/konnectivity-agent-b64jh" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737801 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737832 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737852 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysctl-d\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737862 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91208431-95bb-4e24-ab4b-e0f80f0d6207-tmp\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737831 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737889 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysctl-conf\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.737861 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-systemd-units\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738005 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-var-lib-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738024 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c892baf2-ce51-4c47-8d79-f75f4f50c09d-agent-certs\") pod \"konnectivity-agent-b64jh\" (UID: \"c892baf2-ce51-4c47-8d79-f75f4f50c09d\") " pod="kube-system/konnectivity-agent-b64jh" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738036 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738055 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738066 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-hosts-file\") pod \"node-resolver-wvff6\" (UID: \"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa\") " pod="openshift-dns/node-resolver-wvff6" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738053 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-secret-metrics-server-client-certs\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738094 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7b81351-0c71-49b6-bd86-e48483b8b555-kubelet-config\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738124 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-kubelet\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738156 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738191 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40c70642-d50d-4f92-8ca1-10015c1a40cd-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738196 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738220 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738245 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738262 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51dbc54b-218b-4274-9ebb-4cdbab400d22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738272 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-config-out\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738232 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-iptables-alerter-script\") pod \"iptables-alerter-sm6rp\" (UID: \"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0\") " pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738325 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-etc-sysctl-conf\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738357 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b2eb2a42-2573-414d-8516-aac46f430550-audit-log\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.738363 5939 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738363 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738394 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.738407 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:49.23839268 +0000 UTC m=+1.727667857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-oauth-config" not registered Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738440 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-config-out\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738454 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738468 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-secret-metrics-server-tls\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738493 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-var-lib-openvswitch\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738501 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738521 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91208431-95bb-4e24-ab4b-e0f80f0d6207-tmp\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.738529 5939 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738532 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-socket-dir-parent\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.738592 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:49.238578954 +0000 UTC m=+1.727854121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"service-ca" not registered Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738640 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f94485d7-b1a8-4caa-bfac-39c7cf29bfaa-hosts-file\") pod \"node-resolver-wvff6\" (UID: \"f94485d7-b1a8-4caa-bfac-39c7cf29bfaa\") " pod="openshift-dns/node-resolver-wvff6" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738700 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7e79efa-c2ed-4cc0-928c-e0b9abf5275b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dhwx4\" (UID: \"e7e79efa-c2ed-4cc0-928c-e0b9abf5275b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738732 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86e13c77-adcc-4f59-b1a4-93d090e2379e-metrics-client-ca\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738756 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-tls\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738761 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b2eb2a42-2573-414d-8516-aac46f430550-secret-metrics-server-tls\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738780 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7b81351-0c71-49b6-bd86-e48483b8b555-kubelet-config\") pod \"global-pull-secret-syncer-dvl2d\" (UID: \"e7b81351-0c71-49b6-bd86-e48483b8b555\") " pod="kube-system/global-pull-secret-syncer-dvl2d" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738869 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a8478f3-fe5c-4c04-b0a7-08bb736f3ede-lib-modules\") pod \"tuned-dwxbv\" (UID: \"5a8478f3-fe5c-4c04-b0a7-08bb736f3ede\") " pod="openshift-cluster-node-tuning-operator/tuned-dwxbv" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738906 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40c70642-d50d-4f92-8ca1-10015c1a40cd-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738918 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738955 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86e13c77-adcc-4f59-b1a4-93d090e2379e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.738961 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-config-out\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.739096 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.739133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.739157 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b2eb2a42-2573-414d-8516-aac46f430550-metrics-server-audit-profiles\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:48.740706 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.739220 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-config-out\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.740706 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.739294 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dca6aeb-205b-4634-9216-a98ba3957025-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:29:48.740706 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.739873 5939 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:48.740706 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.740066 5939 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:48.744307 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.744290 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8dl6\" (UniqueName: \"kubernetes.io/projected/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-kube-api-access-q8dl6\") pod \"iptables-alerter-sm6rp\" (UID: \"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0\") " pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:29:48.744429 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.744408 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcmn7\" (UniqueName: \"kubernetes.io/projected/1db4e02b-1e7a-4361-994e-f30507dbf0e7-kube-api-access-gcmn7\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.744755 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.744738 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6j5l\" (UniqueName: \"kubernetes.io/projected/57decfb3-58bb-4b9c-b834-205594071bdf-kube-api-access-v6j5l\") pod \"ingress-canary-pnmtp\" (UID: \"57decfb3-58bb-4b9c-b834-205594071bdf\") " pod="openshift-ingress-canary/ingress-canary-pnmtp" Oct 31 00:29:48.746234 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.746194 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdn2m\" (UniqueName: \"kubernetes.io/projected/79b89acd-a759-4036-bd91-16ee9ad9e288-kube-api-access-mdn2m\") pod \"node-ca-9l52l\" (UID: \"79b89acd-a759-4036-bd91-16ee9ad9e288\") " pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:29:48.746234 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.746207 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6gnm\" (UniqueName: \"kubernetes.io/projected/51dbc54b-218b-4274-9ebb-4cdbab400d22-kube-api-access-h6gnm\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:48.746389 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.746273 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29dqz\" (UniqueName: \"kubernetes.io/projected/266a13d4-3f45-44d6-ac0e-3e171ce4e928-kube-api-access-29dqz\") pod \"monitoring-plugin-77f4f76594-65wl9\" (UID: \"266a13d4-3f45-44d6-ac0e-3e171ce4e928\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" Oct 31 00:29:48.746508 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.746490 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwjlt\" (UniqueName: \"kubernetes.io/projected/b2eb2a42-2573-414d-8516-aac46f430550-kube-api-access-xwjlt\") pod \"metrics-server-67cc7bf54c-zxwf4\" (UID: \"b2eb2a42-2573-414d-8516-aac46f430550\") " pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:48.746613 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.746593 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22g5c\" (UniqueName: \"kubernetes.io/projected/a4677743-13cb-4a3a-b1df-8bd6baef94f4-kube-api-access-22g5c\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.746650 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.746619 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26png\" (UniqueName: \"kubernetes.io/projected/6dca6aeb-205b-4634-9216-a98ba3957025-kube-api-access-26png\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:29:48.747364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.747347 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wvxm\" (UniqueName: \"kubernetes.io/projected/3073e91a-8363-4d2a-8774-635302d4b040-kube-api-access-4wvxm\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.747845 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.747824 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbxs7\" (UniqueName: \"kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-kube-api-access-tbxs7\") pod \"prometheus-k8s-0\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:48.749992 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.749979 5939 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 00:29:48.750043 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.749995 5939 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 00:29:48.750043 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.750009 5939 projected.go:196] Error preparing data for projected volume kube-api-access-4mfhm for pod openshift-console/console-74f876fbf6-4mlmt: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:29:48.750096 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.750057 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:49.25004711 +0000 UTC m=+1.739322266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4mfhm" (UniqueName: "kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:29:48.751526 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.751508 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bthrt\" (UniqueName: \"kubernetes.io/projected/8d06594d-071d-47bb-ade5-16298fd91e2c-kube-api-access-bthrt\") pod \"kube-state-metrics-5dc8d7d89b-mjkzk\" (UID: \"8d06594d-071d-47bb-ade5-16298fd91e2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mjkzk" Oct 31 00:29:48.751973 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.751954 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc9ss\" (UniqueName: \"kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-kube-api-access-sc9ss\") pod \"alertmanager-main-0\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:48.758124 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.758105 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcx7m\" (UniqueName: \"kubernetes.io/projected/40c70642-d50d-4f92-8ca1-10015c1a40cd-kube-api-access-qcx7m\") pod \"openshift-state-metrics-7bd7489957-srmg2\" (UID: \"40c70642-d50d-4f92-8ca1-10015c1a40cd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-srmg2" Oct 31 00:29:48.758807 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.758792 5939 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:29:48.758873 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.758833 5939 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:48.759378 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.759363 5939 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:29:48.759467 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:48.759397 5939 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:29:48.763819 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.763802 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvdnb\" (UniqueName: \"kubernetes.io/projected/37dd17e5-ba62-4f95-bfc7-05b132104e44-kube-api-access-wvdnb\") pod \"dns-default-xkqrm\" (UID: \"37dd17e5-ba62-4f95-bfc7-05b132104e44\") " pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:48.764791 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.764772 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmzx8\" (UniqueName: \"kubernetes.io/projected/86e13c77-adcc-4f59-b1a4-93d090e2379e-kube-api-access-bmzx8\") pod \"thanos-querier-7bcdf957ff-s472c\" (UID: \"86e13c77-adcc-4f59-b1a4-93d090e2379e\") " pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:48.839759 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.839729 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-k8s-cni-cncf-io\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.839759 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.839759 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-cni-bin\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.839928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.839778 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:29:48.839928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.839801 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-etc-kubernetes\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.839928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.839837 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-root\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.839928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.839836 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-k8s-cni-cncf-io\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.839928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.839849 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-cni-bin\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.839928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.839862 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-system-cni-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.839928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.839885 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-etc-kubernetes\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.839928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.839884 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:29:48.839928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.839914 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-system-cni-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.839928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.839912 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-root\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.839936 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-wtmp\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.839957 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-cnibin\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.839975 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-sys\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.839992 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-cni-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.839992 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-cnibin\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840027 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-sys\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840058 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-cni-multus\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840060 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-cni-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840083 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-socket-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840112 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-cni-multus\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840119 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a4677743-13cb-4a3a-b1df-8bd6baef94f4-node-exporter-wtmp\") pod \"node-exporter-vl8l6\" (UID: \"a4677743-13cb-4a3a-b1df-8bd6baef94f4\") " pod="openshift-monitoring/node-exporter-vl8l6" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840115 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-socket-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840128 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-device-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840170 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-netns\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840191 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-netns\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840195 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-registration-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840233 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-registration-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840173 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-device-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840266 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-kubelet\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840310 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-socket-dir-parent\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840326 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-sys-fs\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840354 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-var-lib-kubelet\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840365 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-sys-fs\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840372 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-conf-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840388 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-conf-dir\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840366 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-multus-socket-dir-parent\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.840407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840404 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.841375 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840439 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.841375 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840455 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51dbc54b-218b-4274-9ebb-4cdbab400d22-crio-socket\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:48.841375 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840470 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.841375 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840471 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-hostroot\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.841375 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840494 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-hostroot\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.841375 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840498 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1db4e02b-1e7a-4361-994e-f30507dbf0e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ctvst\" (UID: \"1db4e02b-1e7a-4361-994e-f30507dbf0e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctvst" Oct 31 00:29:48.841375 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840509 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51dbc54b-218b-4274-9ebb-4cdbab400d22-crio-socket\") pod \"insights-runtime-extractor-w49pc\" (UID: \"51dbc54b-218b-4274-9ebb-4cdbab400d22\") " pod="openshift-insights/insights-runtime-extractor-w49pc" Oct 31 00:29:48.841375 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840513 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-system-cni-dir\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:29:48.841375 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840537 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-cnibin\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:29:48.841375 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840542 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-system-cni-dir\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:29:48.841375 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840551 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79b89acd-a759-4036-bd91-16ee9ad9e288-host\") pod \"node-ca-9l52l\" (UID: \"79b89acd-a759-4036-bd91-16ee9ad9e288\") " pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:29:48.841375 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840581 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79b89acd-a759-4036-bd91-16ee9ad9e288-host\") pod \"node-ca-9l52l\" (UID: \"79b89acd-a759-4036-bd91-16ee9ad9e288\") " pod="openshift-image-registry/node-ca-9l52l" Oct 31 00:29:48.841375 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840582 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-multus-certs\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.841375 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840604 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-host-slash\") pod \"iptables-alerter-sm6rp\" (UID: \"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0\") " pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:29:48.841375 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840613 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3073e91a-8363-4d2a-8774-635302d4b040-host-run-multus-certs\") pod \"multus-pghrd\" (UID: \"3073e91a-8363-4d2a-8774-635302d4b040\") " pod="openshift-multus/multus-pghrd" Oct 31 00:29:48.841375 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840613 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dca6aeb-205b-4634-9216-a98ba3957025-cnibin\") pod \"multus-additional-cni-plugins-dcljc\" (UID: \"6dca6aeb-205b-4634-9216-a98ba3957025\") " pod="openshift-multus/multus-additional-cni-plugins-dcljc" Oct 31 00:29:48.841375 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:48.840680 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0-host-slash\") pod \"iptables-alerter-sm6rp\" (UID: \"cd2f6cc8-7a15-44cd-99c1-53ee713c3fb0\") " pod="openshift-network-operator/iptables-alerter-sm6rp" Oct 31 00:29:49.244378 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:49.244337 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:49.244378 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:49.244381 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:49.244602 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:49.244406 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:49.244602 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:49.244424 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:49.244602 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:49.244446 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:49.244602 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:49.244459 5939 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 00:29:49.244602 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:49.244479 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:49.244602 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:49.244481 5939 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:29:49.244602 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:49.244481 5939 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:29:49.244602 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:49.244539 5939 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 00:29:49.244602 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:49.244546 5939 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 00:29:49.244602 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:49.244549 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:50.244530528 +0000 UTC m=+2.733805708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"service-ca" not registered Oct 31 00:29:49.244602 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:49.244585 5939 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 00:29:49.244602 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:49.244607 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:50.244591555 +0000 UTC m=+2.733866712 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:29:49.244984 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:49.244624 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:50.244615026 +0000 UTC m=+2.733890219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:29:49.244984 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:49.244641 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:50.244632769 +0000 UTC m=+2.733907926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-oauth-config" not registered Oct 31 00:29:49.244984 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:49.244656 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:50.244647323 +0000 UTC m=+2.733922480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-config" not registered Oct 31 00:29:49.244984 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:49.244691 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:50.244660941 +0000 UTC m=+2.733936097 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-serving-cert" not registered Oct 31 00:29:49.345597 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:49.345567 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mfhm\" (UniqueName: \"kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:49.345754 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:49.345687 5939 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 00:29:49.345754 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:49.345700 5939 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 00:29:49.345754 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:49.345709 5939 projected.go:196] Error preparing data for projected volume kube-api-access-4mfhm for pod openshift-console/console-74f876fbf6-4mlmt: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:29:49.345754 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:49.345747 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:50.345735171 +0000 UTC m=+2.835010327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4mfhm" (UniqueName: "kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:29:50.254171 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:50.254136 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:50.254171 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:50.254173 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:50.254701 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:50.254199 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:50.254701 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:50.254218 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:50.254701 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:50.254270 5939 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:29:50.254701 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:50.254273 5939 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 00:29:50.254701 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:50.254325 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:52.254312569 +0000 UTC m=+4.743587725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:29:50.254701 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:50.254339 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:50.254701 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:50.254360 5939 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 00:29:50.254701 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:50.254378 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:52.254359524 +0000 UTC m=+4.743634700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"service-ca" not registered Oct 31 00:29:50.254701 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:50.254273 5939 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:29:50.254701 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:50.254378 5939 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 00:29:50.254701 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:50.254415 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:52.254400287 +0000 UTC m=+4.743675460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-oauth-config" not registered Oct 31 00:29:50.254701 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:50.254438 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:52.254428788 +0000 UTC m=+4.743703957 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-config" not registered Oct 31 00:29:50.254701 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:50.254456 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:52.254447658 +0000 UTC m=+4.743723021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:29:50.254701 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:50.254498 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:50.254701 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:50.254619 5939 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 00:29:50.254701 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:50.254696 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:52.254683541 +0000 UTC m=+4.743958719 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-serving-cert" not registered Oct 31 00:29:50.355237 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:50.355204 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mfhm\" (UniqueName: \"kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:50.355376 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:50.355360 5939 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 00:29:50.355414 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:50.355382 5939 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 00:29:50.355414 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:50.355392 5939 projected.go:196] Error preparing data for projected volume kube-api-access-4mfhm for pod openshift-console/console-74f876fbf6-4mlmt: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:29:50.355494 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:50.355440 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:52.355425041 +0000 UTC m=+4.844700212 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4mfhm" (UniqueName: "kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:29:50.704567 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:50.704537 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xkqrm" Oct 31 00:29:51.447316 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:51.447282 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7bcdf957ff-s472c" Oct 31 00:29:52.124477 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:52.124440 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:52.271811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:52.271768 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:52.271811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:52.271821 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:52.272053 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:52.271856 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:52.272053 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:52.271875 5939 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:29:52.272053 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:52.271905 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:52.272053 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:52.271925 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:52.272053 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:52.271954 5939 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 00:29:52.272053 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:52.271964 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:52.272053 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:52.271991 5939 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 00:29:52.272053 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:52.271999 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:56.271972812 +0000 UTC m=+8.761247982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:29:52.272053 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:52.272026 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:56.272014921 +0000 UTC m=+8.761290080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-config" not registered Oct 31 00:29:52.272053 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:52.272028 5939 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:29:52.272053 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:52.272046 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:56.272037087 +0000 UTC m=+8.761312244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-serving-cert" not registered Oct 31 00:29:52.272053 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:52.272047 5939 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 00:29:52.272424 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:52.271991 5939 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 00:29:52.272424 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:52.272078 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:56.272063713 +0000 UTC m=+8.761338877 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:29:52.272424 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:52.272097 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:56.272087167 +0000 UTC m=+8.761362328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-oauth-config" not registered Oct 31 00:29:52.272424 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:52.272108 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:56.27210202 +0000 UTC m=+8.761377176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"service-ca" not registered Oct 31 00:29:52.373249 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:52.373211 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mfhm\" (UniqueName: \"kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:52.373413 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:52.373348 5939 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 00:29:52.373413 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:52.373366 5939 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 00:29:52.373413 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:52.373387 5939 projected.go:196] Error preparing data for projected volume kube-api-access-4mfhm for pod openshift-console/console-74f876fbf6-4mlmt: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:29:52.373563 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:52.373440 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:56.373424099 +0000 UTC m=+8.862699265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4mfhm" (UniqueName: "kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:29:52.884095 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:52.884070 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:55.095394 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:55.095359 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:29:55.095871 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:55.095579 5939 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:29:55.149170 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:55.149141 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:29:55.836713 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:55.836686 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-65wl9" Oct 31 00:29:56.331856 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:56.331822 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:56.332282 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:56.331962 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:56.332282 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:56.331962 5939 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 00:29:56.332282 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:56.331985 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:56.332282 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:56.332011 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:56.332282 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:56.332034 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:30:04.332020213 +0000 UTC m=+16.821295370 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-serving-cert" not registered Oct 31 00:29:56.332282 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:56.332062 5939 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:29:56.332282 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:56.332076 5939 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 00:29:56.332282 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:56.332103 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:30:04.332093452 +0000 UTC m=+16.821368609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:29:56.332282 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:56.332114 5939 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 00:29:56.332282 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:56.332146 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:30:04.332130105 +0000 UTC m=+16.821405280 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"service-ca" not registered Oct 31 00:29:56.332282 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:56.332155 5939 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:29:56.332282 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:56.332167 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:30:04.332155297 +0000 UTC m=+16.821430467 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-oauth-config" not registered Oct 31 00:29:56.332282 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:56.332181 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:30:04.332175577 +0000 UTC m=+16.821450734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:29:56.332282 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:56.332121 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:56.332282 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:56.332218 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:56.332746 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:56.332304 5939 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 00:29:56.332746 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:56.332361 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:30:04.332349854 +0000 UTC m=+16.821625013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-config" not registered Oct 31 00:29:56.337406 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:56.337384 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-67cc7bf54c-zxwf4" Oct 31 00:29:56.433152 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:56.433120 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mfhm\" (UniqueName: \"kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:29:56.433301 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:56.433273 5939 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 00:29:56.433301 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:56.433296 5939 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 00:29:56.433396 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:56.433309 5939 projected.go:196] Error preparing data for projected volume kube-api-access-4mfhm for pod openshift-console/console-74f876fbf6-4mlmt: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:29:56.433396 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:29:56.433386 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:30:04.433367287 +0000 UTC m=+16.922642467 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4mfhm" (UniqueName: "kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:29:57.545998 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:29:57.545965 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dhwx4" Oct 31 00:30:00.019923 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:00.019849 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b64jh" Oct 31 00:30:00.508537 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:00.508501 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:30:01.820511 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:01.820475 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:30:01.821006 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:01.820930 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="alertmanager" containerID="cri-o://0f52f91ae90659018cb48c193137efa09e1b312f780516f36c5c6d53522f8cf7" gracePeriod=120 Oct 31 00:30:01.821006 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:01.820956 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="kube-rbac-proxy" containerID="cri-o://d1259cbc9034b86982202e36778e9c3115b54e49f6a26d09d1b5ff34b05faa8d" gracePeriod=120 Oct 31 00:30:01.821006 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:01.820955 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="kube-rbac-proxy-web" containerID="cri-o://c987cd437379dc739d69eab5766ddbed2c4a612117986fe268fdd2056899ce9c" gracePeriod=120 Oct 31 00:30:01.821185 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:01.820911 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="kube-rbac-proxy-metric" containerID="cri-o://f65e4e8455ee99e23dd7c759b18664c4a505a28fc815a22e8fd101dd01ea654f" gracePeriod=120 Oct 31 00:30:01.821185 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:01.820931 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="prom-label-proxy" containerID="cri-o://cf709572a498967a1d01ac5976ebdc5c977f8ae769a867cfe17ffdde910cb80a" gracePeriod=120 Oct 31 00:30:01.821185 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:01.821082 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="config-reloader" containerID="cri-o://1e800c60b24a1ab552d955413f9f8b2ab2a3540b3d28f27face081d026cddc03" gracePeriod=120 Oct 31 00:30:02.800895 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:02.800865 5939 generic.go:356] "Generic (PLEG): container finished" podID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerID="cf709572a498967a1d01ac5976ebdc5c977f8ae769a867cfe17ffdde910cb80a" exitCode=0 Oct 31 00:30:02.800895 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:02.800888 5939 generic.go:356] "Generic (PLEG): container finished" podID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerID="f65e4e8455ee99e23dd7c759b18664c4a505a28fc815a22e8fd101dd01ea654f" exitCode=0 Oct 31 00:30:02.800895 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:02.800897 5939 generic.go:356] "Generic (PLEG): container finished" podID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerID="d1259cbc9034b86982202e36778e9c3115b54e49f6a26d09d1b5ff34b05faa8d" exitCode=0 Oct 31 00:30:02.800895 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:02.800906 5939 generic.go:356] "Generic (PLEG): container finished" podID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerID="1e800c60b24a1ab552d955413f9f8b2ab2a3540b3d28f27face081d026cddc03" exitCode=0 Oct 31 00:30:02.800895 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:02.800911 5939 generic.go:356] "Generic (PLEG): container finished" podID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerID="0f52f91ae90659018cb48c193137efa09e1b312f780516f36c5c6d53522f8cf7" exitCode=0 Oct 31 00:30:02.801184 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:02.800936 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bfa459c8-39bb-4020-a514-0e03536a3dda","Type":"ContainerDied","Data":"cf709572a498967a1d01ac5976ebdc5c977f8ae769a867cfe17ffdde910cb80a"} Oct 31 00:30:02.801184 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:02.801020 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bfa459c8-39bb-4020-a514-0e03536a3dda","Type":"ContainerDied","Data":"f65e4e8455ee99e23dd7c759b18664c4a505a28fc815a22e8fd101dd01ea654f"} Oct 31 00:30:02.801184 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:02.801047 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bfa459c8-39bb-4020-a514-0e03536a3dda","Type":"ContainerDied","Data":"d1259cbc9034b86982202e36778e9c3115b54e49f6a26d09d1b5ff34b05faa8d"} Oct 31 00:30:02.801184 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:02.801070 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bfa459c8-39bb-4020-a514-0e03536a3dda","Type":"ContainerDied","Data":"1e800c60b24a1ab552d955413f9f8b2ab2a3540b3d28f27face081d026cddc03"} Oct 31 00:30:02.801184 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:02.801094 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bfa459c8-39bb-4020-a514-0e03536a3dda","Type":"ContainerDied","Data":"0f52f91ae90659018cb48c193137efa09e1b312f780516f36c5c6d53522f8cf7"} Oct 31 00:30:03.109887 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.109864 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:03.299706 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.299657 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy\") pod \"bfa459c8-39bb-4020-a514-0e03536a3dda\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " Oct 31 00:30:03.299866 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.299729 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-trusted-ca-bundle\") pod \"bfa459c8-39bb-4020-a514-0e03536a3dda\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " Oct 31 00:30:03.299866 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.299761 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-cluster-tls-config\") pod \"bfa459c8-39bb-4020-a514-0e03536a3dda\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " Oct 31 00:30:03.299866 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.299785 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-config-out\") pod \"bfa459c8-39bb-4020-a514-0e03536a3dda\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " Oct 31 00:30:03.299866 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.299802 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-tls-assets\" (UniqueName: \"kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-tls-assets\") pod \"bfa459c8-39bb-4020-a514-0e03536a3dda\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " Oct 31 00:30:03.299866 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.299820 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-web-config\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-web-config\") pod \"bfa459c8-39bb-4020-a514-0e03536a3dda\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " Oct 31 00:30:03.300099 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.299880 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-metrics-client-ca\") pod \"bfa459c8-39bb-4020-a514-0e03536a3dda\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " Oct 31 00:30:03.300099 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.299916 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-main-db\") pod \"bfa459c8-39bb-4020-a514-0e03536a3dda\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " Oct 31 00:30:03.300099 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.299943 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-kube-api-access-sc9ss\" (UniqueName: \"kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-kube-api-access-sc9ss\") pod \"bfa459c8-39bb-4020-a514-0e03536a3dda\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " Oct 31 00:30:03.300099 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.299973 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-config-volume\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-config-volume\") pod \"bfa459c8-39bb-4020-a514-0e03536a3dda\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " Oct 31 00:30:03.300099 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.300059 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-metric\") pod \"bfa459c8-39bb-4020-a514-0e03536a3dda\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " Oct 31 00:30:03.300381 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.300103 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-main-tls\") pod \"bfa459c8-39bb-4020-a514-0e03536a3dda\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " Oct 31 00:30:03.300381 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.300133 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-web\") pod \"bfa459c8-39bb-4020-a514-0e03536a3dda\" (UID: \"bfa459c8-39bb-4020-a514-0e03536a3dda\") " Oct 31 00:30:03.300381 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.300167 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-trusted-ca-bundle" pod "bfa459c8-39bb-4020-a514-0e03536a3dda" (UID: "bfa459c8-39bb-4020-a514-0e03536a3dda"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:03.300698 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.300542 5939 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:03.300698 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.300598 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-metrics-client-ca" pod "bfa459c8-39bb-4020-a514-0e03536a3dda" (UID: "bfa459c8-39bb-4020-a514-0e03536a3dda"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:03.301161 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.300956 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-main-db" pod "bfa459c8-39bb-4020-a514-0e03536a3dda" (UID: "bfa459c8-39bb-4020-a514-0e03536a3dda"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:03.303198 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.303176 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy" pod "bfa459c8-39bb-4020-a514-0e03536a3dda" (UID: "bfa459c8-39bb-4020-a514-0e03536a3dda"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:03.303304 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.303278 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-kube-api-access-sc9ss" pod "bfa459c8-39bb-4020-a514-0e03536a3dda" (UID: "bfa459c8-39bb-4020-a514-0e03536a3dda"). InnerVolumeSpecName "kube-api-access-sc9ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:03.303561 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.303471 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-tls-assets" pod "bfa459c8-39bb-4020-a514-0e03536a3dda" (UID: "bfa459c8-39bb-4020-a514-0e03536a3dda"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:03.303561 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.303513 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-config-out" pod "bfa459c8-39bb-4020-a514-0e03536a3dda" (UID: "bfa459c8-39bb-4020-a514-0e03536a3dda"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:03.303718 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.303556 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-metric" pod "bfa459c8-39bb-4020-a514-0e03536a3dda" (UID: "bfa459c8-39bb-4020-a514-0e03536a3dda"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:03.303816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.303722 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-web" pod "bfa459c8-39bb-4020-a514-0e03536a3dda" (UID: "bfa459c8-39bb-4020-a514-0e03536a3dda"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:03.304982 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.304962 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-main-tls" pod "bfa459c8-39bb-4020-a514-0e03536a3dda" (UID: "bfa459c8-39bb-4020-a514-0e03536a3dda"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:03.305503 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.305486 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-config-volume" pod "bfa459c8-39bb-4020-a514-0e03536a3dda" (UID: "bfa459c8-39bb-4020-a514-0e03536a3dda"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:03.307759 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.307736 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-cluster-tls-config" pod "bfa459c8-39bb-4020-a514-0e03536a3dda" (UID: "bfa459c8-39bb-4020-a514-0e03536a3dda"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:03.314057 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.314034 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-web-config" pod "bfa459c8-39bb-4020-a514-0e03536a3dda" (UID: "bfa459c8-39bb-4020-a514-0e03536a3dda"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:03.401124 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.401100 5939 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-cluster-tls-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:03.401238 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.401125 5939 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-config-out\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:03.401238 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.401139 5939 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-tls-assets\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:03.401238 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.401152 5939 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-web-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:03.401238 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.401164 5939 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfa459c8-39bb-4020-a514-0e03536a3dda-metrics-client-ca\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:03.401238 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.401177 5939 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bfa459c8-39bb-4020-a514-0e03536a3dda-alertmanager-main-db\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:03.401238 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.401191 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc9ss\" (UniqueName: \"kubernetes.io/projected/bfa459c8-39bb-4020-a514-0e03536a3dda-kube-api-access-sc9ss\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:03.401238 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.401204 5939 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-config-volume\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:03.401238 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.401219 5939 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:03.401238 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.401234 5939 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-main-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:03.401528 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.401248 5939 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:03.401528 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.401263 5939 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfa459c8-39bb-4020-a514-0e03536a3dda-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:03.807349 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.807317 5939 generic.go:356] "Generic (PLEG): container finished" podID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerID="c987cd437379dc739d69eab5766ddbed2c4a612117986fe268fdd2056899ce9c" exitCode=0 Oct 31 00:30:03.807533 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.807374 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bfa459c8-39bb-4020-a514-0e03536a3dda","Type":"ContainerDied","Data":"c987cd437379dc739d69eab5766ddbed2c4a612117986fe268fdd2056899ce9c"} Oct 31 00:30:03.807533 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.807401 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:03.807533 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.807431 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bfa459c8-39bb-4020-a514-0e03536a3dda","Type":"ContainerDied","Data":"acaf343745ccfd10174cdb916b6488c6fe4fb0428d40402e6c863d3e03a9f596"} Oct 31 00:30:03.807533 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.807471 5939 scope.go:122] "RemoveContainer" containerID="cf709572a498967a1d01ac5976ebdc5c977f8ae769a867cfe17ffdde910cb80a" Oct 31 00:30:03.815244 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.815225 5939 scope.go:122] "RemoveContainer" containerID="f65e4e8455ee99e23dd7c759b18664c4a505a28fc815a22e8fd101dd01ea654f" Oct 31 00:30:03.822096 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.822068 5939 scope.go:122] "RemoveContainer" containerID="d1259cbc9034b86982202e36778e9c3115b54e49f6a26d09d1b5ff34b05faa8d" Oct 31 00:30:03.828748 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.828730 5939 scope.go:122] "RemoveContainer" containerID="c987cd437379dc739d69eab5766ddbed2c4a612117986fe268fdd2056899ce9c" Oct 31 00:30:03.834797 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.834777 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:30:03.835636 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.835616 5939 scope.go:122] "RemoveContainer" containerID="1e800c60b24a1ab552d955413f9f8b2ab2a3540b3d28f27face081d026cddc03" Oct 31 00:30:03.842257 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.842240 5939 scope.go:122] "RemoveContainer" containerID="0f52f91ae90659018cb48c193137efa09e1b312f780516f36c5c6d53522f8cf7" Oct 31 00:30:03.847704 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.847684 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:30:03.850006 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.849987 5939 scope.go:122] "RemoveContainer" containerID="fb92b869942ab49a2d6edb574af10bc1fcb0d928852268c7469de8726e518eef" Oct 31 00:30:03.856677 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.856644 5939 scope.go:122] "RemoveContainer" containerID="cf709572a498967a1d01ac5976ebdc5c977f8ae769a867cfe17ffdde910cb80a" Oct 31 00:30:03.856920 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:03.856901 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf709572a498967a1d01ac5976ebdc5c977f8ae769a867cfe17ffdde910cb80a\": container with ID starting with cf709572a498967a1d01ac5976ebdc5c977f8ae769a867cfe17ffdde910cb80a not found: ID does not exist" containerID="cf709572a498967a1d01ac5976ebdc5c977f8ae769a867cfe17ffdde910cb80a" Oct 31 00:30:03.856973 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.856933 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf709572a498967a1d01ac5976ebdc5c977f8ae769a867cfe17ffdde910cb80a"} err="failed to get container status \"cf709572a498967a1d01ac5976ebdc5c977f8ae769a867cfe17ffdde910cb80a\": rpc error: code = NotFound desc = could not find container \"cf709572a498967a1d01ac5976ebdc5c977f8ae769a867cfe17ffdde910cb80a\": container with ID starting with cf709572a498967a1d01ac5976ebdc5c977f8ae769a867cfe17ffdde910cb80a not found: ID does not exist" Oct 31 00:30:03.856973 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.856970 5939 scope.go:122] "RemoveContainer" containerID="f65e4e8455ee99e23dd7c759b18664c4a505a28fc815a22e8fd101dd01ea654f" Oct 31 00:30:03.857208 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:03.857191 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f65e4e8455ee99e23dd7c759b18664c4a505a28fc815a22e8fd101dd01ea654f\": container with ID starting with f65e4e8455ee99e23dd7c759b18664c4a505a28fc815a22e8fd101dd01ea654f not found: ID does not exist" containerID="f65e4e8455ee99e23dd7c759b18664c4a505a28fc815a22e8fd101dd01ea654f" Oct 31 00:30:03.857258 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.857213 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65e4e8455ee99e23dd7c759b18664c4a505a28fc815a22e8fd101dd01ea654f"} err="failed to get container status \"f65e4e8455ee99e23dd7c759b18664c4a505a28fc815a22e8fd101dd01ea654f\": rpc error: code = NotFound desc = could not find container \"f65e4e8455ee99e23dd7c759b18664c4a505a28fc815a22e8fd101dd01ea654f\": container with ID starting with f65e4e8455ee99e23dd7c759b18664c4a505a28fc815a22e8fd101dd01ea654f not found: ID does not exist" Oct 31 00:30:03.857258 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.857232 5939 scope.go:122] "RemoveContainer" containerID="d1259cbc9034b86982202e36778e9c3115b54e49f6a26d09d1b5ff34b05faa8d" Oct 31 00:30:03.857480 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:03.857466 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1259cbc9034b86982202e36778e9c3115b54e49f6a26d09d1b5ff34b05faa8d\": container with ID starting with d1259cbc9034b86982202e36778e9c3115b54e49f6a26d09d1b5ff34b05faa8d not found: ID does not exist" containerID="d1259cbc9034b86982202e36778e9c3115b54e49f6a26d09d1b5ff34b05faa8d" Oct 31 00:30:03.857537 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.857484 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1259cbc9034b86982202e36778e9c3115b54e49f6a26d09d1b5ff34b05faa8d"} err="failed to get container status \"d1259cbc9034b86982202e36778e9c3115b54e49f6a26d09d1b5ff34b05faa8d\": rpc error: code = NotFound desc = could not find container \"d1259cbc9034b86982202e36778e9c3115b54e49f6a26d09d1b5ff34b05faa8d\": container with ID starting with d1259cbc9034b86982202e36778e9c3115b54e49f6a26d09d1b5ff34b05faa8d not found: ID does not exist" Oct 31 00:30:03.857537 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.857511 5939 scope.go:122] "RemoveContainer" containerID="c987cd437379dc739d69eab5766ddbed2c4a612117986fe268fdd2056899ce9c" Oct 31 00:30:03.857741 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:03.857727 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c987cd437379dc739d69eab5766ddbed2c4a612117986fe268fdd2056899ce9c\": container with ID starting with c987cd437379dc739d69eab5766ddbed2c4a612117986fe268fdd2056899ce9c not found: ID does not exist" containerID="c987cd437379dc739d69eab5766ddbed2c4a612117986fe268fdd2056899ce9c" Oct 31 00:30:03.857777 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.857745 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c987cd437379dc739d69eab5766ddbed2c4a612117986fe268fdd2056899ce9c"} err="failed to get container status \"c987cd437379dc739d69eab5766ddbed2c4a612117986fe268fdd2056899ce9c\": rpc error: code = NotFound desc = could not find container \"c987cd437379dc739d69eab5766ddbed2c4a612117986fe268fdd2056899ce9c\": container with ID starting with c987cd437379dc739d69eab5766ddbed2c4a612117986fe268fdd2056899ce9c not found: ID does not exist" Oct 31 00:30:03.857777 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.857759 5939 scope.go:122] "RemoveContainer" containerID="1e800c60b24a1ab552d955413f9f8b2ab2a3540b3d28f27face081d026cddc03" Oct 31 00:30:03.857968 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:03.857953 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e800c60b24a1ab552d955413f9f8b2ab2a3540b3d28f27face081d026cddc03\": container with ID starting with 1e800c60b24a1ab552d955413f9f8b2ab2a3540b3d28f27face081d026cddc03 not found: ID does not exist" containerID="1e800c60b24a1ab552d955413f9f8b2ab2a3540b3d28f27face081d026cddc03" Oct 31 00:30:03.858008 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.857972 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e800c60b24a1ab552d955413f9f8b2ab2a3540b3d28f27face081d026cddc03"} err="failed to get container status \"1e800c60b24a1ab552d955413f9f8b2ab2a3540b3d28f27face081d026cddc03\": rpc error: code = NotFound desc = could not find container \"1e800c60b24a1ab552d955413f9f8b2ab2a3540b3d28f27face081d026cddc03\": container with ID starting with 1e800c60b24a1ab552d955413f9f8b2ab2a3540b3d28f27face081d026cddc03 not found: ID does not exist" Oct 31 00:30:03.858008 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.857983 5939 scope.go:122] "RemoveContainer" containerID="0f52f91ae90659018cb48c193137efa09e1b312f780516f36c5c6d53522f8cf7" Oct 31 00:30:03.858202 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:03.858185 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f52f91ae90659018cb48c193137efa09e1b312f780516f36c5c6d53522f8cf7\": container with ID starting with 0f52f91ae90659018cb48c193137efa09e1b312f780516f36c5c6d53522f8cf7 not found: ID does not exist" containerID="0f52f91ae90659018cb48c193137efa09e1b312f780516f36c5c6d53522f8cf7" Oct 31 00:30:03.858250 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.858205 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f52f91ae90659018cb48c193137efa09e1b312f780516f36c5c6d53522f8cf7"} err="failed to get container status \"0f52f91ae90659018cb48c193137efa09e1b312f780516f36c5c6d53522f8cf7\": rpc error: code = NotFound desc = could not find container \"0f52f91ae90659018cb48c193137efa09e1b312f780516f36c5c6d53522f8cf7\": container with ID starting with 0f52f91ae90659018cb48c193137efa09e1b312f780516f36c5c6d53522f8cf7 not found: ID does not exist" Oct 31 00:30:03.858250 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.858217 5939 scope.go:122] "RemoveContainer" containerID="fb92b869942ab49a2d6edb574af10bc1fcb0d928852268c7469de8726e518eef" Oct 31 00:30:03.858430 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:03.858414 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb92b869942ab49a2d6edb574af10bc1fcb0d928852268c7469de8726e518eef\": container with ID starting with fb92b869942ab49a2d6edb574af10bc1fcb0d928852268c7469de8726e518eef not found: ID does not exist" containerID="fb92b869942ab49a2d6edb574af10bc1fcb0d928852268c7469de8726e518eef" Oct 31 00:30:03.858473 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.858433 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb92b869942ab49a2d6edb574af10bc1fcb0d928852268c7469de8726e518eef"} err="failed to get container status \"fb92b869942ab49a2d6edb574af10bc1fcb0d928852268c7469de8726e518eef\": rpc error: code = NotFound desc = could not find container \"fb92b869942ab49a2d6edb574af10bc1fcb0d928852268c7469de8726e518eef\": container with ID starting with fb92b869942ab49a2d6edb574af10bc1fcb0d928852268c7469de8726e518eef not found: ID does not exist" Oct 31 00:30:03.883293 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883270 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:30:03.883544 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883532 5939 cpu_manager.go:414] "containerMap: removing container" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="config-reloader" Oct 31 00:30:03.883588 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883545 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="config-reloader" Oct 31 00:30:03.883588 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883558 5939 cpu_manager.go:414] "containerMap: removing container" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="kube-rbac-proxy-web" Oct 31 00:30:03.883588 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883563 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="kube-rbac-proxy-web" Oct 31 00:30:03.883588 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883571 5939 cpu_manager.go:414] "containerMap: removing container" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="init-config-reloader" Oct 31 00:30:03.883588 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883575 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="init-config-reloader" Oct 31 00:30:03.883588 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883584 5939 cpu_manager.go:414] "containerMap: removing container" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="kube-rbac-proxy" Oct 31 00:30:03.883588 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883589 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="kube-rbac-proxy" Oct 31 00:30:03.883827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883599 5939 cpu_manager.go:414] "containerMap: removing container" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="prom-label-proxy" Oct 31 00:30:03.883827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883604 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="prom-label-proxy" Oct 31 00:30:03.883827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883611 5939 cpu_manager.go:414] "containerMap: removing container" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="alertmanager" Oct 31 00:30:03.883827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883615 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="alertmanager" Oct 31 00:30:03.883827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883626 5939 cpu_manager.go:414] "containerMap: removing container" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="kube-rbac-proxy-metric" Oct 31 00:30:03.883827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883631 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="kube-rbac-proxy-metric" Oct 31 00:30:03.883827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883691 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="kube-rbac-proxy-metric" Oct 31 00:30:03.883827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883700 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="config-reloader" Oct 31 00:30:03.883827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883706 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="kube-rbac-proxy-web" Oct 31 00:30:03.883827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883712 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="init-config-reloader" Oct 31 00:30:03.883827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883716 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="kube-rbac-proxy" Oct 31 00:30:03.883827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883722 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="prom-label-proxy" Oct 31 00:30:03.883827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.883726 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" containerName="alertmanager" Oct 31 00:30:03.886735 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.886714 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:03.897459 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.897441 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:30:03.904604 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.904586 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:30:03.904604 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.904598 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:30:03.904741 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.904612 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:30:03.904741 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.904603 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:30:03.904884 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.904858 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:30:03.904884 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.904875 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:30:03.905031 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.904901 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:30:03.905031 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.904928 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:30:03.905031 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.905006 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9gnrx\"" Oct 31 00:30:03.907804 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:03.907761 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:30:04.007853 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.007823 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.007994 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.007859 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.007994 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.007879 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-config-volume\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.008067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.008008 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.008067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.008050 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4498092-a01a-4bad-b4a3-7197685a3272-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.008127 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.008077 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.008180 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.008166 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4498092-a01a-4bad-b4a3-7197685a3272-config-out\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.008215 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.008191 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4498092-a01a-4bad-b4a3-7197685a3272-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.008253 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.008219 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.008350 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.008322 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phxkc\" (UniqueName: \"kubernetes.io/projected/c4498092-a01a-4bad-b4a3-7197685a3272-kube-api-access-phxkc\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.008394 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.008379 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-web-config\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.008436 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.008424 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4498092-a01a-4bad-b4a3-7197685a3272-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.008469 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.008445 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4498092-a01a-4bad-b4a3-7197685a3272-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.109730 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.109698 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4498092-a01a-4bad-b4a3-7197685a3272-config-out\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.109888 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.109733 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4498092-a01a-4bad-b4a3-7197685a3272-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.109888 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.109771 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.109888 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.109800 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phxkc\" (UniqueName: \"kubernetes.io/projected/c4498092-a01a-4bad-b4a3-7197685a3272-kube-api-access-phxkc\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.109888 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.109845 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-web-config\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.109888 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.109866 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4498092-a01a-4bad-b4a3-7197685a3272-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.110440 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.109993 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4498092-a01a-4bad-b4a3-7197685a3272-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.110440 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.110069 5939 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:30:04.110440 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.110071 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.110440 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.110214 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.110440 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.110245 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4498092-a01a-4bad-b4a3-7197685a3272-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.110440 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.110247 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-config-volume\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.110440 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.110340 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.110440 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.110394 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4498092-a01a-4bad-b4a3-7197685a3272-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.110440 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.110437 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.110911 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.110566 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4498092-a01a-4bad-b4a3-7197685a3272-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.111011 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.110987 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4498092-a01a-4bad-b4a3-7197685a3272-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.113039 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.112712 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.113202 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.113174 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.113312 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.113234 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.113376 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.113359 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4498092-a01a-4bad-b4a3-7197685a3272-config-out\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.113428 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.113406 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4498092-a01a-4bad-b4a3-7197685a3272-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.113610 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.113586 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.113856 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.113835 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-config-volume\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.114351 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.114328 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-web-config\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.115160 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.115141 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4498092-a01a-4bad-b4a3-7197685a3272-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.125012 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.124991 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phxkc\" (UniqueName: \"kubernetes.io/projected/c4498092-a01a-4bad-b4a3-7197685a3272-kube-api-access-phxkc\") pod \"alertmanager-main-0\" (UID: \"c4498092-a01a-4bad-b4a3-7197685a3272\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.202938 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.202879 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:30:04.414139 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.414109 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:30:04.414139 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.414139 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:30:04.414364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.414167 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:30:04.414364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.414193 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:30:04.414364 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:04.414222 5939 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 00:30:04.414364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.414231 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:30:04.414364 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:04.414242 5939 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:30:04.414364 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:04.414260 5939 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 00:30:04.414364 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:04.414278 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:30:20.414265188 +0000 UTC m=+32.903540344 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"service-ca" not registered Oct 31 00:30:04.414364 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:04.414296 5939 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 00:30:04.414364 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:04.414314 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:30:20.414300019 +0000 UTC m=+32.903575179 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:30:04.414364 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:04.414326 5939 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:30:04.414364 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:04.414337 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:30:20.41432954 +0000 UTC m=+32.903604729 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-oauth-config" not registered Oct 31 00:30:04.414364 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:04.414355 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:30:20.414347307 +0000 UTC m=+32.903622491 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-config" not registered Oct 31 00:30:04.414805 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.414405 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:30:04.414805 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:04.414423 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:30:20.414400815 +0000 UTC m=+32.903675988 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:30:04.414805 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:04.414479 5939 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 00:30:04.414805 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:04.414530 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:30:20.414518584 +0000 UTC m=+32.903793759 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : object "openshift-console"/"console-serving-cert" not registered Oct 31 00:30:04.492656 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.492584 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:30:04.497528 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:30:04.497504 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4498092_a01a_4bad_b4a3_7197685a3272.slice/crio-d6294ee8541c978c17b0a16a441ce06c4b5c20278551374435c4dc1d23fdbe72 WatchSource:0}: Error finding container d6294ee8541c978c17b0a16a441ce06c4b5c20278551374435c4dc1d23fdbe72: Status 404 returned error can't find the container with id d6294ee8541c978c17b0a16a441ce06c4b5c20278551374435c4dc1d23fdbe72 Oct 31 00:30:04.516787 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.516343 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mfhm\" (UniqueName: \"kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm\") pod \"console-74f876fbf6-4mlmt\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:30:04.517720 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:04.516903 5939 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 00:30:04.517720 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:04.516938 5939 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 00:30:04.517720 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:04.516952 5939 projected.go:196] Error preparing data for projected volume kube-api-access-4mfhm for pod openshift-console/console-74f876fbf6-4mlmt: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:30:04.517720 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:04.517019 5939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm podName:91208431-95bb-4e24-ab4b-e0f80f0d6207 nodeName:}" failed. No retries permitted until 2026-10-31 00:30:20.517000346 +0000 UTC m=+33.006275516 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4mfhm" (UniqueName: "kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm") pod "console-74f876fbf6-4mlmt" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:30:04.813369 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.813271 5939 generic.go:356] "Generic (PLEG): container finished" podID="c4498092-a01a-4bad-b4a3-7197685a3272" containerID="3c9152848768ea5b2af4dd895cb367837725adb1471c5e4ce29ff421445227e2" exitCode=0 Oct 31 00:30:04.813369 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.813351 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4498092-a01a-4bad-b4a3-7197685a3272","Type":"ContainerDied","Data":"3c9152848768ea5b2af4dd895cb367837725adb1471c5e4ce29ff421445227e2"} Oct 31 00:30:04.813567 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:04.813503 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4498092-a01a-4bad-b4a3-7197685a3272","Type":"ContainerStarted","Data":"d6294ee8541c978c17b0a16a441ce06c4b5c20278551374435c4dc1d23fdbe72"} Oct 31 00:30:05.653525 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.653493 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa459c8-39bb-4020-a514-0e03536a3dda" path="/var/lib/kubelet/pods/bfa459c8-39bb-4020-a514-0e03536a3dda/volumes" Oct 31 00:30:05.824718 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.824661 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4498092-a01a-4bad-b4a3-7197685a3272","Type":"ContainerStarted","Data":"7b18b854dcc6b7e82e3294c1ead04f1f334ad25a680ef55cac3d28d2c4950256"} Oct 31 00:30:05.824876 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.824752 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4498092-a01a-4bad-b4a3-7197685a3272","Type":"ContainerStarted","Data":"d77d16fcc92f579100b9c90d226ddf021014a1d50e2ac3c7d8ccf61bf60d7d2f"} Oct 31 00:30:05.824876 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.824784 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4498092-a01a-4bad-b4a3-7197685a3272","Type":"ContainerStarted","Data":"ccb57220cb7f90c8d41e8f164eb8dc8a6147416191f9b30a131c163c6f745ecb"} Oct 31 00:30:05.824876 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.824810 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4498092-a01a-4bad-b4a3-7197685a3272","Type":"ContainerStarted","Data":"56607b2474aa547bf1e2753d47aa5a320429c4dc85e40cf58d497b00bc4dd377"} Oct 31 00:30:05.824876 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.824832 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4498092-a01a-4bad-b4a3-7197685a3272","Type":"ContainerStarted","Data":"b6859f7e5403b0eaf24363ae178e7602c5be2d3dfa748d2879f66cca19e74922"} Oct 31 00:30:05.824876 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.824854 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4498092-a01a-4bad-b4a3-7197685a3272","Type":"ContainerStarted","Data":"0317d0fc5ab94d174b38a9ed6bfce0d93c976c45673665d181b7ab05f3f9ab7d"} Oct 31 00:30:05.867852 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.867799 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.855136147 podStartE2EDuration="2.867785215s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.649068ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:05.867344757 +0000 UTC m=+18.356619938" watchObservedRunningTime="2026-10-31 00:30:05.867785215 +0000 UTC m=+18.357060427" Oct 31 00:30:05.876478 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.876449 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-766d564798-98prp"] Oct 31 00:30:05.879208 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.879186 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:05.901015 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.900984 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-766d564798-98prp"] Oct 31 00:30:05.930090 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.929950 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6e2db5-ca96-4448-8eb5-65e33c760582-serving-certs-ca-bundle\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:05.930090 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.929981 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6a6e2db5-ca96-4448-8eb5-65e33c760582-federate-client-tls\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:05.930416 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.930382 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a6e2db5-ca96-4448-8eb5-65e33c760582-metrics-client-ca\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:05.930900 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.930775 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6e2db5-ca96-4448-8eb5-65e33c760582-telemeter-trusted-ca-bundle\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:05.931543 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.931519 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6a6e2db5-ca96-4448-8eb5-65e33c760582-telemeter-client-tls\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:05.931637 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.931572 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a6e2db5-ca96-4448-8eb5-65e33c760582-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:05.931733 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.931705 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6a6e2db5-ca96-4448-8eb5-65e33c760582-secret-telemeter-client\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:05.931795 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.931756 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc479\" (UniqueName: \"kubernetes.io/projected/6a6e2db5-ca96-4448-8eb5-65e33c760582-kube-api-access-lc479\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:05.947259 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.947237 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 31 00:30:05.947377 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.947360 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 31 00:30:05.947487 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.947471 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 31 00:30:05.947521 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.947500 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fvb7q\"" Oct 31 00:30:05.947593 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.947578 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 31 00:30:05.976786 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.976768 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 31 00:30:05.979184 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:05.979169 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 31 00:30:06.033627 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.033595 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6a6e2db5-ca96-4448-8eb5-65e33c760582-telemeter-client-tls\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:06.033751 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.033633 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a6e2db5-ca96-4448-8eb5-65e33c760582-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:06.033751 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.033660 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6a6e2db5-ca96-4448-8eb5-65e33c760582-secret-telemeter-client\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:06.033751 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.033711 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc479\" (UniqueName: \"kubernetes.io/projected/6a6e2db5-ca96-4448-8eb5-65e33c760582-kube-api-access-lc479\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:06.033903 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.033810 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6e2db5-ca96-4448-8eb5-65e33c760582-serving-certs-ca-bundle\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:06.033903 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.033834 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6a6e2db5-ca96-4448-8eb5-65e33c760582-federate-client-tls\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:06.034336 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.033959 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a6e2db5-ca96-4448-8eb5-65e33c760582-metrics-client-ca\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:06.034336 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.034006 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6e2db5-ca96-4448-8eb5-65e33c760582-telemeter-trusted-ca-bundle\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:06.034644 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.034615 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6e2db5-ca96-4448-8eb5-65e33c760582-serving-certs-ca-bundle\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:06.034963 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.034940 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a6e2db5-ca96-4448-8eb5-65e33c760582-metrics-client-ca\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:06.035132 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.035107 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6e2db5-ca96-4448-8eb5-65e33c760582-telemeter-trusted-ca-bundle\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:06.036458 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.036439 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6a6e2db5-ca96-4448-8eb5-65e33c760582-federate-client-tls\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:06.036458 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.036445 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6a6e2db5-ca96-4448-8eb5-65e33c760582-secret-telemeter-client\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:06.036575 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.036555 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a6e2db5-ca96-4448-8eb5-65e33c760582-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:06.036575 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.036566 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6a6e2db5-ca96-4448-8eb5-65e33c760582-telemeter-client-tls\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:06.048876 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.048852 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc479\" (UniqueName: \"kubernetes.io/projected/6a6e2db5-ca96-4448-8eb5-65e33c760582-kube-api-access-lc479\") pod \"telemeter-client-766d564798-98prp\" (UID: \"6a6e2db5-ca96-4448-8eb5-65e33c760582\") " pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:06.193543 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.193476 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-766d564798-98prp" Oct 31 00:30:06.214987 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.214965 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:30:06.215438 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.215246 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="prometheus" containerID="cri-o://f9af75c3923d6d5d6b484625c5f9200ac1099246e2d32f96fbbfbe767f49b930" gracePeriod=600 Oct 31 00:30:06.215438 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.215270 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="kube-rbac-proxy" containerID="cri-o://30600e3dbe12f6b65c49307713f4f1628da379e8962f6faa812f3993e402e2c8" gracePeriod=600 Oct 31 00:30:06.215438 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.215301 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="thanos-sidecar" containerID="cri-o://81cac64659d6aa22472fe0bbbf50f05d1b9d5b9b2b1290a60883f4fe9401aa6b" gracePeriod=600 Oct 31 00:30:06.215438 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.215327 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="kube-rbac-proxy-web" containerID="cri-o://5f183a8c26cf96890f3cf614f2f1425529ff8f38aeb87c475440d1277fde8e16" gracePeriod=600 Oct 31 00:30:06.215438 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.215327 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e5db3ac7f06883e8365d3b6d52e6ee8ac52f98637c43bbe56640fe0b32fe6ad8" gracePeriod=600 Oct 31 00:30:06.215438 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.215328 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="config-reloader" containerID="cri-o://93356c90d027466eb63fd46907afa38a144ab942bcf1276f1fedb82aba60c243" gracePeriod=600 Oct 31 00:30:06.486034 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.485970 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-766d564798-98prp"] Oct 31 00:30:06.488797 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:30:06.488775 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6e2db5_ca96_4448_8eb5_65e33c760582.slice/crio-d486629c6827b32c3bbd45054626e1bf31f832f5262af30e0e68010ba3a3b4d6 WatchSource:0}: Error finding container d486629c6827b32c3bbd45054626e1bf31f832f5262af30e0e68010ba3a3b4d6: Status 404 returned error can't find the container with id d486629c6827b32c3bbd45054626e1bf31f832f5262af30e0e68010ba3a3b4d6 Oct 31 00:30:06.490652 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.490634 5939 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:30:06.829712 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.829602 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766d564798-98prp" event={"ID":"6a6e2db5-ca96-4448-8eb5-65e33c760582","Type":"ContainerStarted","Data":"d486629c6827b32c3bbd45054626e1bf31f832f5262af30e0e68010ba3a3b4d6"} Oct 31 00:30:06.832929 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.832902 5939 generic.go:356] "Generic (PLEG): container finished" podID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerID="e5db3ac7f06883e8365d3b6d52e6ee8ac52f98637c43bbe56640fe0b32fe6ad8" exitCode=0 Oct 31 00:30:06.832929 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.832925 5939 generic.go:356] "Generic (PLEG): container finished" podID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerID="30600e3dbe12f6b65c49307713f4f1628da379e8962f6faa812f3993e402e2c8" exitCode=0 Oct 31 00:30:06.833060 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.832934 5939 generic.go:356] "Generic (PLEG): container finished" podID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerID="81cac64659d6aa22472fe0bbbf50f05d1b9d5b9b2b1290a60883f4fe9401aa6b" exitCode=0 Oct 31 00:30:06.833060 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.832941 5939 generic.go:356] "Generic (PLEG): container finished" podID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerID="93356c90d027466eb63fd46907afa38a144ab942bcf1276f1fedb82aba60c243" exitCode=0 Oct 31 00:30:06.833060 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.832947 5939 generic.go:356] "Generic (PLEG): container finished" podID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerID="f9af75c3923d6d5d6b484625c5f9200ac1099246e2d32f96fbbfbe767f49b930" exitCode=0 Oct 31 00:30:06.833060 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.832977 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffc16559-1d24-40ea-9ba6-aa6896178faf","Type":"ContainerDied","Data":"e5db3ac7f06883e8365d3b6d52e6ee8ac52f98637c43bbe56640fe0b32fe6ad8"} Oct 31 00:30:06.833060 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.833032 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffc16559-1d24-40ea-9ba6-aa6896178faf","Type":"ContainerDied","Data":"30600e3dbe12f6b65c49307713f4f1628da379e8962f6faa812f3993e402e2c8"} Oct 31 00:30:06.833060 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.833060 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffc16559-1d24-40ea-9ba6-aa6896178faf","Type":"ContainerDied","Data":"81cac64659d6aa22472fe0bbbf50f05d1b9d5b9b2b1290a60883f4fe9401aa6b"} Oct 31 00:30:06.833244 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.833085 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffc16559-1d24-40ea-9ba6-aa6896178faf","Type":"ContainerDied","Data":"93356c90d027466eb63fd46907afa38a144ab942bcf1276f1fedb82aba60c243"} Oct 31 00:30:06.833244 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:06.833107 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffc16559-1d24-40ea-9ba6-aa6896178faf","Type":"ContainerDied","Data":"f9af75c3923d6d5d6b484625c5f9200ac1099246e2d32f96fbbfbe767f49b930"} Oct 31 00:30:07.475470 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.475445 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.549863 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.549488 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-metrics-client-certs\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.549863 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.549531 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-kube-api-access-tbxs7\" (UniqueName: \"kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-kube-api-access-tbxs7\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.549863 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.549712 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-config\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-config\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.549863 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.549778 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-0\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.549863 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.549847 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.550229 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.549893 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-1\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.550229 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.549924 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-config-out\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.550229 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.549956 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-tls-assets\" (UniqueName: \"kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-tls-assets\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.550229 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.549999 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-tls\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.550229 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.550024 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-grpc-tls\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.550229 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.550050 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.550229 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.550096 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-2\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.550229 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.550139 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-kube-rbac-proxy\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.550229 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.550172 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-web-config\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-web-config\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.550229 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.550206 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-thanos-prometheus-http-client-file\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.550772 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.550237 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-trusted-ca-bundle\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.550772 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.550268 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-metrics-client-ca\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.550772 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.550309 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-serving-certs-ca-bundle\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.550772 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.550352 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-db\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.550772 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.550385 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-kubelet-serving-ca-bundle\") pod \"ffc16559-1d24-40ea-9ba6-aa6896178faf\" (UID: \"ffc16559-1d24-40ea-9ba6-aa6896178faf\") " Oct 31 00:30:07.551938 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.551644 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-trusted-ca-bundle" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:07.552460 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.552193 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-metrics-client-ca" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:07.552460 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.552332 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-kubelet-serving-ca-bundle" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:07.552609 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.552563 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-serving-certs-ca-bundle" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:07.553105 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.552843 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-config" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:07.553532 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.553472 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-db" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:07.554633 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.554612 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-2" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:07.554949 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.554926 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-1" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:07.555360 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.555335 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-0" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:07.555716 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.555641 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-kube-api-access-tbxs7" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "kube-api-access-tbxs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:07.555716 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.555645 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-metrics-client-certs" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:07.556019 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.555952 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-tls-assets" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:07.557082 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.557001 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-kube-rbac-proxy" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:07.557486 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.557464 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-grpc-tls" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:07.557651 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.557626 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-thanos-sidecar-tls" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:07.558146 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.558093 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-kube-rbac-proxy-web" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:07.558247 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.558225 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-thanos-prometheus-http-client-file" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:07.558317 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.558267 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-config-out" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:07.559655 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.559629 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-tls" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:07.572215 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.572180 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-web-config" pod "ffc16559-1d24-40ea-9ba6-aa6896178faf" (UID: "ffc16559-1d24-40ea-9ba6-aa6896178faf"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:07.651555 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651526 5939 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651555 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651557 5939 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651572 5939 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651586 5939 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-config-out\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651600 5939 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-tls-assets\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651612 5939 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651625 5939 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-grpc-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651637 5939 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651651 5939 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651677 5939 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-kube-rbac-proxy\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651691 5939 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-web-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651704 5939 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651718 5939 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651731 5939 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-metrics-client-ca\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651744 5939 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651757 5939 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffc16559-1d24-40ea-9ba6-aa6896178faf-prometheus-k8s-db\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651781 5939 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc16559-1d24-40ea-9ba6-aa6896178faf-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651796 5939 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-secret-metrics-client-certs\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651810 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbxs7\" (UniqueName: \"kubernetes.io/projected/ffc16559-1d24-40ea-9ba6-aa6896178faf-kube-api-access-tbxs7\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.651816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.651824 5939 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ffc16559-1d24-40ea-9ba6-aa6896178faf-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:07.849940 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.849852 5939 generic.go:356] "Generic (PLEG): container finished" podID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerID="5f183a8c26cf96890f3cf614f2f1425529ff8f38aeb87c475440d1277fde8e16" exitCode=0 Oct 31 00:30:07.849940 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.849908 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffc16559-1d24-40ea-9ba6-aa6896178faf","Type":"ContainerDied","Data":"5f183a8c26cf96890f3cf614f2f1425529ff8f38aeb87c475440d1277fde8e16"} Oct 31 00:30:07.850395 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.849989 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.850395 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.850015 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffc16559-1d24-40ea-9ba6-aa6896178faf","Type":"ContainerDied","Data":"c45165966107cab5eea0f3a82ed97e285ea82de8829878df0a678cee5634d91c"} Oct 31 00:30:07.850395 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.850054 5939 scope.go:122] "RemoveContainer" containerID="e5db3ac7f06883e8365d3b6d52e6ee8ac52f98637c43bbe56640fe0b32fe6ad8" Oct 31 00:30:07.862134 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.862111 5939 scope.go:122] "RemoveContainer" containerID="30600e3dbe12f6b65c49307713f4f1628da379e8962f6faa812f3993e402e2c8" Oct 31 00:30:07.870279 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.870258 5939 scope.go:122] "RemoveContainer" containerID="5f183a8c26cf96890f3cf614f2f1425529ff8f38aeb87c475440d1277fde8e16" Oct 31 00:30:07.878971 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.878944 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:30:07.879806 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.879785 5939 scope.go:122] "RemoveContainer" containerID="81cac64659d6aa22472fe0bbbf50f05d1b9d5b9b2b1290a60883f4fe9401aa6b" Oct 31 00:30:07.886849 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.886830 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:30:07.888133 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.888113 5939 scope.go:122] "RemoveContainer" containerID="93356c90d027466eb63fd46907afa38a144ab942bcf1276f1fedb82aba60c243" Oct 31 00:30:07.896319 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.896290 5939 scope.go:122] "RemoveContainer" containerID="f9af75c3923d6d5d6b484625c5f9200ac1099246e2d32f96fbbfbe767f49b930" Oct 31 00:30:07.904962 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.904944 5939 scope.go:122] "RemoveContainer" containerID="eb74ac6c958294b3fdf7f5e08d3c3921e026e9c566c85820664a00f8705ccb49" Oct 31 00:30:07.912510 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.912489 5939 scope.go:122] "RemoveContainer" containerID="e5db3ac7f06883e8365d3b6d52e6ee8ac52f98637c43bbe56640fe0b32fe6ad8" Oct 31 00:30:07.913009 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:07.912975 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5db3ac7f06883e8365d3b6d52e6ee8ac52f98637c43bbe56640fe0b32fe6ad8\": container with ID starting with e5db3ac7f06883e8365d3b6d52e6ee8ac52f98637c43bbe56640fe0b32fe6ad8 not found: ID does not exist" containerID="e5db3ac7f06883e8365d3b6d52e6ee8ac52f98637c43bbe56640fe0b32fe6ad8" Oct 31 00:30:07.913100 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.913012 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5db3ac7f06883e8365d3b6d52e6ee8ac52f98637c43bbe56640fe0b32fe6ad8"} err="failed to get container status \"e5db3ac7f06883e8365d3b6d52e6ee8ac52f98637c43bbe56640fe0b32fe6ad8\": rpc error: code = NotFound desc = could not find container \"e5db3ac7f06883e8365d3b6d52e6ee8ac52f98637c43bbe56640fe0b32fe6ad8\": container with ID starting with e5db3ac7f06883e8365d3b6d52e6ee8ac52f98637c43bbe56640fe0b32fe6ad8 not found: ID does not exist" Oct 31 00:30:07.913100 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.913037 5939 scope.go:122] "RemoveContainer" containerID="30600e3dbe12f6b65c49307713f4f1628da379e8962f6faa812f3993e402e2c8" Oct 31 00:30:07.913335 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:07.913316 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30600e3dbe12f6b65c49307713f4f1628da379e8962f6faa812f3993e402e2c8\": container with ID starting with 30600e3dbe12f6b65c49307713f4f1628da379e8962f6faa812f3993e402e2c8 not found: ID does not exist" containerID="30600e3dbe12f6b65c49307713f4f1628da379e8962f6faa812f3993e402e2c8" Oct 31 00:30:07.913417 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.913340 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30600e3dbe12f6b65c49307713f4f1628da379e8962f6faa812f3993e402e2c8"} err="failed to get container status \"30600e3dbe12f6b65c49307713f4f1628da379e8962f6faa812f3993e402e2c8\": rpc error: code = NotFound desc = could not find container \"30600e3dbe12f6b65c49307713f4f1628da379e8962f6faa812f3993e402e2c8\": container with ID starting with 30600e3dbe12f6b65c49307713f4f1628da379e8962f6faa812f3993e402e2c8 not found: ID does not exist" Oct 31 00:30:07.913417 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.913361 5939 scope.go:122] "RemoveContainer" containerID="5f183a8c26cf96890f3cf614f2f1425529ff8f38aeb87c475440d1277fde8e16" Oct 31 00:30:07.913639 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:07.913610 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f183a8c26cf96890f3cf614f2f1425529ff8f38aeb87c475440d1277fde8e16\": container with ID starting with 5f183a8c26cf96890f3cf614f2f1425529ff8f38aeb87c475440d1277fde8e16 not found: ID does not exist" containerID="5f183a8c26cf96890f3cf614f2f1425529ff8f38aeb87c475440d1277fde8e16" Oct 31 00:30:07.913746 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.913645 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f183a8c26cf96890f3cf614f2f1425529ff8f38aeb87c475440d1277fde8e16"} err="failed to get container status \"5f183a8c26cf96890f3cf614f2f1425529ff8f38aeb87c475440d1277fde8e16\": rpc error: code = NotFound desc = could not find container \"5f183a8c26cf96890f3cf614f2f1425529ff8f38aeb87c475440d1277fde8e16\": container with ID starting with 5f183a8c26cf96890f3cf614f2f1425529ff8f38aeb87c475440d1277fde8e16 not found: ID does not exist" Oct 31 00:30:07.913746 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.913700 5939 scope.go:122] "RemoveContainer" containerID="81cac64659d6aa22472fe0bbbf50f05d1b9d5b9b2b1290a60883f4fe9401aa6b" Oct 31 00:30:07.913974 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:07.913958 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81cac64659d6aa22472fe0bbbf50f05d1b9d5b9b2b1290a60883f4fe9401aa6b\": container with ID starting with 81cac64659d6aa22472fe0bbbf50f05d1b9d5b9b2b1290a60883f4fe9401aa6b not found: ID does not exist" containerID="81cac64659d6aa22472fe0bbbf50f05d1b9d5b9b2b1290a60883f4fe9401aa6b" Oct 31 00:30:07.914035 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.913979 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81cac64659d6aa22472fe0bbbf50f05d1b9d5b9b2b1290a60883f4fe9401aa6b"} err="failed to get container status \"81cac64659d6aa22472fe0bbbf50f05d1b9d5b9b2b1290a60883f4fe9401aa6b\": rpc error: code = NotFound desc = could not find container \"81cac64659d6aa22472fe0bbbf50f05d1b9d5b9b2b1290a60883f4fe9401aa6b\": container with ID starting with 81cac64659d6aa22472fe0bbbf50f05d1b9d5b9b2b1290a60883f4fe9401aa6b not found: ID does not exist" Oct 31 00:30:07.914035 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.913991 5939 scope.go:122] "RemoveContainer" containerID="93356c90d027466eb63fd46907afa38a144ab942bcf1276f1fedb82aba60c243" Oct 31 00:30:07.914342 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:07.914319 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93356c90d027466eb63fd46907afa38a144ab942bcf1276f1fedb82aba60c243\": container with ID starting with 93356c90d027466eb63fd46907afa38a144ab942bcf1276f1fedb82aba60c243 not found: ID does not exist" containerID="93356c90d027466eb63fd46907afa38a144ab942bcf1276f1fedb82aba60c243" Oct 31 00:30:07.914420 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.914345 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93356c90d027466eb63fd46907afa38a144ab942bcf1276f1fedb82aba60c243"} err="failed to get container status \"93356c90d027466eb63fd46907afa38a144ab942bcf1276f1fedb82aba60c243\": rpc error: code = NotFound desc = could not find container \"93356c90d027466eb63fd46907afa38a144ab942bcf1276f1fedb82aba60c243\": container with ID starting with 93356c90d027466eb63fd46907afa38a144ab942bcf1276f1fedb82aba60c243 not found: ID does not exist" Oct 31 00:30:07.914420 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.914358 5939 scope.go:122] "RemoveContainer" containerID="f9af75c3923d6d5d6b484625c5f9200ac1099246e2d32f96fbbfbe767f49b930" Oct 31 00:30:07.914603 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:07.914586 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9af75c3923d6d5d6b484625c5f9200ac1099246e2d32f96fbbfbe767f49b930\": container with ID starting with f9af75c3923d6d5d6b484625c5f9200ac1099246e2d32f96fbbfbe767f49b930 not found: ID does not exist" containerID="f9af75c3923d6d5d6b484625c5f9200ac1099246e2d32f96fbbfbe767f49b930" Oct 31 00:30:07.914706 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.914610 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9af75c3923d6d5d6b484625c5f9200ac1099246e2d32f96fbbfbe767f49b930"} err="failed to get container status \"f9af75c3923d6d5d6b484625c5f9200ac1099246e2d32f96fbbfbe767f49b930\": rpc error: code = NotFound desc = could not find container \"f9af75c3923d6d5d6b484625c5f9200ac1099246e2d32f96fbbfbe767f49b930\": container with ID starting with f9af75c3923d6d5d6b484625c5f9200ac1099246e2d32f96fbbfbe767f49b930 not found: ID does not exist" Oct 31 00:30:07.914706 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.914629 5939 scope.go:122] "RemoveContainer" containerID="eb74ac6c958294b3fdf7f5e08d3c3921e026e9c566c85820664a00f8705ccb49" Oct 31 00:30:07.915201 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:07.915176 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb74ac6c958294b3fdf7f5e08d3c3921e026e9c566c85820664a00f8705ccb49\": container with ID starting with eb74ac6c958294b3fdf7f5e08d3c3921e026e9c566c85820664a00f8705ccb49 not found: ID does not exist" containerID="eb74ac6c958294b3fdf7f5e08d3c3921e026e9c566c85820664a00f8705ccb49" Oct 31 00:30:07.915278 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.915203 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb74ac6c958294b3fdf7f5e08d3c3921e026e9c566c85820664a00f8705ccb49"} err="failed to get container status \"eb74ac6c958294b3fdf7f5e08d3c3921e026e9c566c85820664a00f8705ccb49\": rpc error: code = NotFound desc = could not find container \"eb74ac6c958294b3fdf7f5e08d3c3921e026e9c566c85820664a00f8705ccb49\": container with ID starting with eb74ac6c958294b3fdf7f5e08d3c3921e026e9c566c85820664a00f8705ccb49 not found: ID does not exist" Oct 31 00:30:07.923312 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.923290 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:30:07.923739 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.923722 5939 cpu_manager.go:414] "containerMap: removing container" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="init-config-reloader" Oct 31 00:30:07.923739 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.923740 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="init-config-reloader" Oct 31 00:30:07.923864 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.923758 5939 cpu_manager.go:414] "containerMap: removing container" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="kube-rbac-proxy-web" Oct 31 00:30:07.923864 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.923766 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="kube-rbac-proxy-web" Oct 31 00:30:07.923864 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.923783 5939 cpu_manager.go:414] "containerMap: removing container" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="kube-rbac-proxy" Oct 31 00:30:07.923864 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.923788 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="kube-rbac-proxy" Oct 31 00:30:07.923864 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.923807 5939 cpu_manager.go:414] "containerMap: removing container" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="config-reloader" Oct 31 00:30:07.923864 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.923814 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="config-reloader" Oct 31 00:30:07.923864 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.923838 5939 cpu_manager.go:414] "containerMap: removing container" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="kube-rbac-proxy-thanos" Oct 31 00:30:07.923864 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.923847 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="kube-rbac-proxy-thanos" Oct 31 00:30:07.923864 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.923857 5939 cpu_manager.go:414] "containerMap: removing container" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="thanos-sidecar" Oct 31 00:30:07.923864 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.923864 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="thanos-sidecar" Oct 31 00:30:07.924233 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.923878 5939 cpu_manager.go:414] "containerMap: removing container" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="prometheus" Oct 31 00:30:07.924233 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.923885 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="prometheus" Oct 31 00:30:07.924233 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.923955 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="config-reloader" Oct 31 00:30:07.924233 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.923969 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="kube-rbac-proxy-thanos" Oct 31 00:30:07.924233 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.924007 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="thanos-sidecar" Oct 31 00:30:07.924233 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.924018 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="prometheus" Oct 31 00:30:07.924233 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.924037 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="init-config-reloader" Oct 31 00:30:07.924233 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.924044 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="kube-rbac-proxy-web" Oct 31 00:30:07.924233 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.924050 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" containerName="kube-rbac-proxy" Oct 31 00:30:07.927398 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.927381 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.934236 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.934212 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:30:07.934400 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.934369 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:30:07.934508 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.934371 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:30:07.934508 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.934444 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:30:07.934508 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.934465 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:30:07.934508 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.934495 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:30:07.934726 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.934540 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:30:07.934726 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.934647 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:30:07.934726 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.934723 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:30:07.934871 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.934740 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:30:07.934871 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.934858 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-84jxj\"" Oct 31 00:30:07.935097 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.935056 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5q9q9rm8kcsp5\"" Oct 31 00:30:07.935181 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.935140 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:30:07.935181 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.935149 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:30:07.939911 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.939892 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:30:07.940298 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.940280 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:30:07.951242 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.951218 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:30:07.955656 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.955612 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.955764 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.955687 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.955764 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.955716 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvhz8\" (UniqueName: \"kubernetes.io/projected/e9279c14-f83e-412e-acb9-6ec7859541f5-kube-api-access-mvhz8\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.955863 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.955839 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e9279c14-f83e-412e-acb9-6ec7859541f5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.955966 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.955948 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9279c14-f83e-412e-acb9-6ec7859541f5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.956023 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.955971 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.956092 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.956072 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.956146 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.956122 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.956209 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.956192 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.956276 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.956258 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.956352 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.956337 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-config\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.956446 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.956431 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.956500 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.956461 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.956555 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.956529 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.956608 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.956576 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.956657 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.956631 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.956725 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.956688 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.956787 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.956729 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9279c14-f83e-412e-acb9-6ec7859541f5-config-out\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.956787 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.956750 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:07.956787 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:07.956776 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-web-config\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.057972 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.057947 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058100 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.057993 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058100 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058032 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9279c14-f83e-412e-acb9-6ec7859541f5-config-out\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058100 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058054 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058100 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058087 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-web-config\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058302 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058121 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058302 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058173 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058302 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058198 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvhz8\" (UniqueName: \"kubernetes.io/projected/e9279c14-f83e-412e-acb9-6ec7859541f5-kube-api-access-mvhz8\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058302 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058227 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e9279c14-f83e-412e-acb9-6ec7859541f5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058302 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058270 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9279c14-f83e-412e-acb9-6ec7859541f5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058543 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058313 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058543 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058379 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058543 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058425 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058543 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058458 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058543 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058495 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058771 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058560 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-config\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058771 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058598 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058771 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058626 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.058771 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058655 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.059001 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.058970 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.059499 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.059280 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.059705 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.059658 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.059820 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.059732 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.059880 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.059816 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.061061 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.060463 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.061061 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.060909 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e9279c14-f83e-412e-acb9-6ec7859541f5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.061520 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.061500 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.062145 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.061647 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.062145 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.061705 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9279c14-f83e-412e-acb9-6ec7859541f5-config-out\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.062145 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.062078 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-web-config\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.063109 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.063037 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.063394 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.063365 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.063550 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.063528 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9279c14-f83e-412e-acb9-6ec7859541f5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.063984 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.063963 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.064235 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.064213 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.064721 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.064702 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e9279c14-f83e-412e-acb9-6ec7859541f5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.065052 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.065036 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.065459 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.065444 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-config\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.066891 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.066862 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e9279c14-f83e-412e-acb9-6ec7859541f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.074238 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.074183 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvhz8\" (UniqueName: \"kubernetes.io/projected/e9279c14-f83e-412e-acb9-6ec7859541f5-kube-api-access-mvhz8\") pod \"prometheus-k8s-0\" (UID: \"e9279c14-f83e-412e-acb9-6ec7859541f5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.246644 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.246621 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:08.541999 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.541908 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:30:08.544705 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:30:08.544657 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9279c14_f83e_412e_acb9_6ec7859541f5.slice/crio-107ea7ff8217465a3123ad5e3810b379c7f56fa982cff1b377957ff1abcc3304 WatchSource:0}: Error finding container 107ea7ff8217465a3123ad5e3810b379c7f56fa982cff1b377957ff1abcc3304: Status 404 returned error can't find the container with id 107ea7ff8217465a3123ad5e3810b379c7f56fa982cff1b377957ff1abcc3304 Oct 31 00:30:08.855874 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.855788 5939 generic.go:356] "Generic (PLEG): container finished" podID="e9279c14-f83e-412e-acb9-6ec7859541f5" containerID="c0071dc48910973a1b3d23293e1371f1aec3f8fb90f7c54c9c45664d980f39d4" exitCode=0 Oct 31 00:30:08.856264 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.855878 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9279c14-f83e-412e-acb9-6ec7859541f5","Type":"ContainerDied","Data":"c0071dc48910973a1b3d23293e1371f1aec3f8fb90f7c54c9c45664d980f39d4"} Oct 31 00:30:08.856264 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.855955 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9279c14-f83e-412e-acb9-6ec7859541f5","Type":"ContainerStarted","Data":"107ea7ff8217465a3123ad5e3810b379c7f56fa982cff1b377957ff1abcc3304"} Oct 31 00:30:08.858656 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.858632 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766d564798-98prp" event={"ID":"6a6e2db5-ca96-4448-8eb5-65e33c760582","Type":"ContainerStarted","Data":"a039a5dc96c1a170a5bb08eba7da2723d619b82a89bf2e136ec62602a815f22c"} Oct 31 00:30:08.861408 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.861370 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766d564798-98prp" event={"ID":"6a6e2db5-ca96-4448-8eb5-65e33c760582","Type":"ContainerStarted","Data":"da06a8a0adb0b89eefb575e8bb41c4ecc02cd0843c13849f62035d1848cdaa2c"} Oct 31 00:30:08.861510 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.861419 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766d564798-98prp" event={"ID":"6a6e2db5-ca96-4448-8eb5-65e33c760582","Type":"ContainerStarted","Data":"110e562fb3e18d6d8588451ee9931a18853fd2504ab1a1201b0c488e19cdab84"} Oct 31 00:30:08.928399 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:08.928353 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-766d564798-98prp" podStartSLOduration=2.238015649 podStartE2EDuration="3.928339686s" totalImagesPullingTime="1.690324037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:08.927149672 +0000 UTC m=+21.416424853" watchObservedRunningTime="2026-10-31 00:30:08.928339686 +0000 UTC m=+21.417614874" Oct 31 00:30:09.651818 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:09.651783 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc16559-1d24-40ea-9ba6-aa6896178faf" path="/var/lib/kubelet/pods/ffc16559-1d24-40ea-9ba6-aa6896178faf/volumes" Oct 31 00:30:09.869658 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:09.869620 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9279c14-f83e-412e-acb9-6ec7859541f5","Type":"ContainerStarted","Data":"62b3a319687c4eb1fe025360ba4673285006201ef8d4dc7ea0016b14d71a2f5d"} Oct 31 00:30:09.870053 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:09.869725 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9279c14-f83e-412e-acb9-6ec7859541f5","Type":"ContainerStarted","Data":"580b74c85a58e2c54184a66d83993a19ef5c7a3b5be0123841d519c402fb8647"} Oct 31 00:30:09.870053 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:09.869771 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9279c14-f83e-412e-acb9-6ec7859541f5","Type":"ContainerStarted","Data":"00578d4806dc50833d2df58b7ffb3f5510d7e34d8f3f75d72f2b0af68ffe6627"} Oct 31 00:30:09.870053 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:09.869814 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9279c14-f83e-412e-acb9-6ec7859541f5","Type":"ContainerStarted","Data":"1a2eb71c22b8adf0d4bde47bd33fd91b09b3d77520b595adad89401e6a71cb40"} Oct 31 00:30:09.870053 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:09.869851 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9279c14-f83e-412e-acb9-6ec7859541f5","Type":"ContainerStarted","Data":"a3ebc8e8964488d4cc8b872d1e07e8a9a033753e8f32a329a163c070e1643e57"} Oct 31 00:30:09.870053 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:09.869887 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9279c14-f83e-412e-acb9-6ec7859541f5","Type":"ContainerStarted","Data":"51484b4e12ea803051d9cc5452b6b3fca3a64cacef775220404fe66bf2c32606"} Oct 31 00:30:09.919487 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:09.919402 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.9062576829999998 podStartE2EDuration="2.919388879s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.131196ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:09.9182454 +0000 UTC m=+22.407520602" watchObservedRunningTime="2026-10-31 00:30:09.919388879 +0000 UTC m=+22.408664055" Oct 31 00:30:13.247539 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:13.247502 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:13.638528 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:13.638485 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-74f876fbf6-4mlmt" podUID="91208431-95bb-4e24-ab4b-e0f80f0d6207" containerName="console" containerID="cri-o://01da56d3986382bf6399373b4f396d7863771f24c726394f1f379840f707417d" gracePeriod=15 Oct 31 00:30:13.886220 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:13.886191 5939 generic.go:356] "Generic (PLEG): container finished" podID="91208431-95bb-4e24-ab4b-e0f80f0d6207" containerID="01da56d3986382bf6399373b4f396d7863771f24c726394f1f379840f707417d" exitCode=0 Oct 31 00:30:13.886345 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:13.886261 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f876fbf6-4mlmt" event={"ID":"91208431-95bb-4e24-ab4b-e0f80f0d6207","Type":"ContainerDied","Data":"01da56d3986382bf6399373b4f396d7863771f24c726394f1f379840f707417d"} Oct 31 00:30:13.886345 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:13.886316 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f876fbf6-4mlmt" event={"ID":"91208431-95bb-4e24-ab4b-e0f80f0d6207","Type":"ContainerDied","Data":"91572e8ab1e6067a5d2f4f9a1e81b1859c2d37d3048dda1a2a4aa345af7b7b2a"} Oct 31 00:30:13.886345 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:13.886337 5939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91572e8ab1e6067a5d2f4f9a1e81b1859c2d37d3048dda1a2a4aa345af7b7b2a" Oct 31 00:30:13.898453 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:13.898402 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:30:14.022438 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.022403 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert\") pod \"91208431-95bb-4e24-ab4b-e0f80f0d6207\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " Oct 31 00:30:14.022620 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.022454 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91208431-95bb-4e24-ab4b-e0f80f0d6207-tmp\" (UniqueName: \"kubernetes.io/empty-dir/91208431-95bb-4e24-ab4b-e0f80f0d6207-tmp\") pod \"91208431-95bb-4e24-ab4b-e0f80f0d6207\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " Oct 31 00:30:14.022620 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.022498 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle\") pod \"91208431-95bb-4e24-ab4b-e0f80f0d6207\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " Oct 31 00:30:14.022620 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.022517 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config\") pod \"91208431-95bb-4e24-ab4b-e0f80f0d6207\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " Oct 31 00:30:14.022620 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.022542 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config\") pod \"91208431-95bb-4e24-ab4b-e0f80f0d6207\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " Oct 31 00:30:14.022620 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.022566 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm\" (UniqueName: \"kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm\") pod \"91208431-95bb-4e24-ab4b-e0f80f0d6207\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " Oct 31 00:30:14.022900 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.022842 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91208431-95bb-4e24-ab4b-e0f80f0d6207-tmp" pod "91208431-95bb-4e24-ab4b-e0f80f0d6207" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:14.022957 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.022922 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert" pod "91208431-95bb-4e24-ab4b-e0f80f0d6207" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:14.022957 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.022937 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert\") pod \"91208431-95bb-4e24-ab4b-e0f80f0d6207\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " Oct 31 00:30:14.023056 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.023010 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle" pod "91208431-95bb-4e24-ab4b-e0f80f0d6207" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:14.023056 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.023019 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config" pod "91208431-95bb-4e24-ab4b-e0f80f0d6207" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:14.023056 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.023017 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca\") pod \"91208431-95bb-4e24-ab4b-e0f80f0d6207\" (UID: \"91208431-95bb-4e24-ab4b-e0f80f0d6207\") " Oct 31 00:30:14.023343 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.023315 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca" pod "91208431-95bb-4e24-ab4b-e0f80f0d6207" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:14.023343 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.023340 5939 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-oauth-serving-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.023536 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.023362 5939 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91208431-95bb-4e24-ab4b-e0f80f0d6207-tmp\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.023536 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.023378 5939 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-trusted-ca-bundle\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.023536 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.023393 5939 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.025161 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.025119 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config" pod "91208431-95bb-4e24-ab4b-e0f80f0d6207" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:14.025271 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.025199 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm" pod "91208431-95bb-4e24-ab4b-e0f80f0d6207" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207"). InnerVolumeSpecName "kube-api-access-4mfhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:14.025271 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.025199 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert" pod "91208431-95bb-4e24-ab4b-e0f80f0d6207" (UID: "91208431-95bb-4e24-ab4b-e0f80f0d6207"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:14.124549 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.124513 5939 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-oauth-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.124549 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.124544 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mfhm\" (UniqueName: \"kubernetes.io/projected/91208431-95bb-4e24-ab4b-e0f80f0d6207-kube-api-access-4mfhm\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.124549 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.124555 5939 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91208431-95bb-4e24-ab4b-e0f80f0d6207-console-serving-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.124796 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.124565 5939 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91208431-95bb-4e24-ab4b-e0f80f0d6207-service-ca\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.893169 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.893137 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f876fbf6-4mlmt" Oct 31 00:30:14.918242 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.918213 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74f876fbf6-4mlmt"] Oct 31 00:30:14.926415 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:14.926390 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74f876fbf6-4mlmt"] Oct 31 00:30:15.152185 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:15.152110 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zwhbr" Oct 31 00:30:15.655676 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:15.655639 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91208431-95bb-4e24-ab4b-e0f80f0d6207" path="/var/lib/kubelet/pods/91208431-95bb-4e24-ab4b-e0f80f0d6207/volumes" Oct 31 00:30:30.509111 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:30:30.509069 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:30:36.690891 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:36.690842 5939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.706545 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:36.706522 5939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:36.998026 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:30:36.997952 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:31:00.509397 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:31:00.509359 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:31:30.509365 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:31:30.509274 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:32:00.512416 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:32:00.512375 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:32:30.509778 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:32:30.509743 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:33:00.509713 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:33:00.509599 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:33:30.508404 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:33:30.508351 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:34:00.508904 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:34:00.508867 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:34:30.508573 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:34:30.508532 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:34:47.687994 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:34:47.687965 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:34:47.692857 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:34:47.692840 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:35:00.508427 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:35:00.508382 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:35:30.509167 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:35:30.509130 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:35:47.657180 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:35:47.657153 5939 scope.go:122] "RemoveContainer" containerID="01da56d3986382bf6399373b4f396d7863771f24c726394f1f379840f707417d" Oct 31 00:36:00.508693 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:36:00.508587 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:36:30.508731 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:36:30.508695 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:37:00.509043 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:37:00.509001 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:37:30.510307 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:37:30.510269 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:37:57.520793 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.520759 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-init-mgjt5"] Oct 31 00:37:57.521252 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.521087 5939 cpu_manager.go:414] "containerMap: removing container" podUID="91208431-95bb-4e24-ab4b-e0f80f0d6207" containerName="console" Oct 31 00:37:57.521252 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.521098 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91208431-95bb-4e24-ab4b-e0f80f0d6207" containerName="console" Oct 31 00:37:57.521252 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.521159 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="91208431-95bb-4e24-ab4b-e0f80f0d6207" containerName="console" Oct 31 00:37:57.523876 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.523858 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mgjt5" Oct 31 00:37:57.530742 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.530296 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Oct 31 00:37:57.530742 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.530507 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Oct 31 00:37:57.530742 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.530581 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Oct 31 00:37:57.530742 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.530724 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\"" Oct 31 00:37:57.530902 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.530783 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-f8d7d\"" Oct 31 00:37:57.535717 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.535629 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-mgjt5"] Oct 31 00:37:57.555862 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.555376 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krndk\" (UniqueName: \"kubernetes.io/projected/266dfcf3-3908-4a9c-bb3c-c45b9407bc0c-kube-api-access-krndk\") pod \"s3-init-mgjt5\" (UID: \"266dfcf3-3908-4a9c-bb3c-c45b9407bc0c\") " pod="kserve/s3-init-mgjt5" Oct 31 00:37:57.555862 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.555442 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/266dfcf3-3908-4a9c-bb3c-c45b9407bc0c-model-cache\") pod \"s3-init-mgjt5\" (UID: \"266dfcf3-3908-4a9c-bb3c-c45b9407bc0c\") " pod="kserve/s3-init-mgjt5" Oct 31 00:37:57.656240 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.656211 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krndk\" (UniqueName: \"kubernetes.io/projected/266dfcf3-3908-4a9c-bb3c-c45b9407bc0c-kube-api-access-krndk\") pod \"s3-init-mgjt5\" (UID: \"266dfcf3-3908-4a9c-bb3c-c45b9407bc0c\") " pod="kserve/s3-init-mgjt5" Oct 31 00:37:57.656420 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.656256 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/266dfcf3-3908-4a9c-bb3c-c45b9407bc0c-model-cache\") pod \"s3-init-mgjt5\" (UID: \"266dfcf3-3908-4a9c-bb3c-c45b9407bc0c\") " pod="kserve/s3-init-mgjt5" Oct 31 00:37:57.656767 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.656749 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/266dfcf3-3908-4a9c-bb3c-c45b9407bc0c-model-cache\") pod \"s3-init-mgjt5\" (UID: \"266dfcf3-3908-4a9c-bb3c-c45b9407bc0c\") " pod="kserve/s3-init-mgjt5" Oct 31 00:37:57.672529 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.672497 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krndk\" (UniqueName: \"kubernetes.io/projected/266dfcf3-3908-4a9c-bb3c-c45b9407bc0c-kube-api-access-krndk\") pod \"s3-init-mgjt5\" (UID: \"266dfcf3-3908-4a9c-bb3c-c45b9407bc0c\") " pod="kserve/s3-init-mgjt5" Oct 31 00:37:57.856937 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:57.856842 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mgjt5" Oct 31 00:37:58.134515 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:58.134486 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-mgjt5"] Oct 31 00:37:58.139703 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:37:58.139658 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod266dfcf3_3908_4a9c_bb3c_c45b9407bc0c.slice/crio-9fdb9e4ff99883dde40927ddad46ea4811e31e16e99a0ec0ef068fdca40b8eb7 WatchSource:0}: Error finding container 9fdb9e4ff99883dde40927ddad46ea4811e31e16e99a0ec0ef068fdca40b8eb7: Status 404 returned error can't find the container with id 9fdb9e4ff99883dde40927ddad46ea4811e31e16e99a0ec0ef068fdca40b8eb7 Oct 31 00:37:58.141572 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:58.141557 5939 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:37:58.588598 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:37:58.588512 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mgjt5" event={"ID":"266dfcf3-3908-4a9c-bb3c-c45b9407bc0c","Type":"ContainerStarted","Data":"9fdb9e4ff99883dde40927ddad46ea4811e31e16e99a0ec0ef068fdca40b8eb7"} Oct 31 00:38:00.511980 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:38:00.511929 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:38:02.607692 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:38:02.607626 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mgjt5" event={"ID":"266dfcf3-3908-4a9c-bb3c-c45b9407bc0c","Type":"ContainerStarted","Data":"39527e8b47ca719fe3838468b85057842b5297fcb9fe53dfde11b2eb9a4ef68b"} Oct 31 00:38:03.613041 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:38:03.613006 5939 generic.go:356] "Generic (PLEG): container finished" podID="266dfcf3-3908-4a9c-bb3c-c45b9407bc0c" containerID="39527e8b47ca719fe3838468b85057842b5297fcb9fe53dfde11b2eb9a4ef68b" exitCode=0 Oct 31 00:38:03.613476 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:38:03.613077 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mgjt5" event={"ID":"266dfcf3-3908-4a9c-bb3c-c45b9407bc0c","Type":"ContainerDied","Data":"39527e8b47ca719fe3838468b85057842b5297fcb9fe53dfde11b2eb9a4ef68b"} Oct 31 00:38:08.642581 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:38:08.642542 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mgjt5" event={"ID":"266dfcf3-3908-4a9c-bb3c-c45b9407bc0c","Type":"ContainerStarted","Data":"7435cd613fbb6b5dbcafe903c6c204e4be0d1bcb2107f0ce593524d38d270079"} Oct 31 00:38:08.666293 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:38:08.666240 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-init-mgjt5" podStartSLOduration=2.43250136 podStartE2EDuration="11.666228581s" totalImagesPullingTime="7.870033807s" totalInitContainerRuntime="1.363693414s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:08.66537372 +0000 UTC m=+501.154648899" watchObservedRunningTime="2026-10-31 00:38:08.666228581 +0000 UTC m=+501.155503759" Oct 31 00:38:30.508740 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:38:30.508703 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:39:00.509622 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:39:00.509492 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:39:13.875807 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:13.875775 5939 generic.go:356] "Generic (PLEG): container finished" podID="266dfcf3-3908-4a9c-bb3c-c45b9407bc0c" containerID="7435cd613fbb6b5dbcafe903c6c204e4be0d1bcb2107f0ce593524d38d270079" exitCode=0 Oct 31 00:39:13.876221 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:13.875865 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mgjt5" event={"ID":"266dfcf3-3908-4a9c-bb3c-c45b9407bc0c","Type":"ContainerDied","Data":"7435cd613fbb6b5dbcafe903c6c204e4be0d1bcb2107f0ce593524d38d270079"} Oct 31 00:39:14.999229 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:14.999205 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mgjt5" Oct 31 00:39:15.018253 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:15.018227 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/266dfcf3-3908-4a9c-bb3c-c45b9407bc0c-kube-api-access-krndk\" (UniqueName: \"kubernetes.io/projected/266dfcf3-3908-4a9c-bb3c-c45b9407bc0c-kube-api-access-krndk\") pod \"266dfcf3-3908-4a9c-bb3c-c45b9407bc0c\" (UID: \"266dfcf3-3908-4a9c-bb3c-c45b9407bc0c\") " Oct 31 00:39:15.018344 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:15.018271 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/266dfcf3-3908-4a9c-bb3c-c45b9407bc0c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/266dfcf3-3908-4a9c-bb3c-c45b9407bc0c-model-cache\") pod \"266dfcf3-3908-4a9c-bb3c-c45b9407bc0c\" (UID: \"266dfcf3-3908-4a9c-bb3c-c45b9407bc0c\") " Oct 31 00:39:15.020285 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:15.020247 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266dfcf3-3908-4a9c-bb3c-c45b9407bc0c-kube-api-access-krndk" pod "266dfcf3-3908-4a9c-bb3c-c45b9407bc0c" (UID: "266dfcf3-3908-4a9c-bb3c-c45b9407bc0c"). InnerVolumeSpecName "kube-api-access-krndk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:15.038170 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:15.038143 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266dfcf3-3908-4a9c-bb3c-c45b9407bc0c-model-cache" pod "266dfcf3-3908-4a9c-bb3c-c45b9407bc0c" (UID: "266dfcf3-3908-4a9c-bb3c-c45b9407bc0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:39:15.118791 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:15.118764 5939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/266dfcf3-3908-4a9c-bb3c-c45b9407bc0c-model-cache\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:15.118791 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:15.118788 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krndk\" (UniqueName: \"kubernetes.io/projected/266dfcf3-3908-4a9c-bb3c-c45b9407bc0c-kube-api-access-krndk\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:15.885898 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:15.885863 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mgjt5" event={"ID":"266dfcf3-3908-4a9c-bb3c-c45b9407bc0c","Type":"ContainerDied","Data":"9fdb9e4ff99883dde40927ddad46ea4811e31e16e99a0ec0ef068fdca40b8eb7"} Oct 31 00:39:15.885898 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:15.885904 5939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fdb9e4ff99883dde40927ddad46ea4811e31e16e99a0ec0ef068fdca40b8eb7" Oct 31 00:39:15.886101 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:15.885909 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mgjt5" Oct 31 00:39:17.140463 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.140430 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-clr8b"] Oct 31 00:39:17.140856 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.140774 5939 cpu_manager.go:414] "containerMap: removing container" podUID="266dfcf3-3908-4a9c-bb3c-c45b9407bc0c" containerName="download-hf-model" Oct 31 00:39:17.140856 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.140786 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="266dfcf3-3908-4a9c-bb3c-c45b9407bc0c" containerName="download-hf-model" Oct 31 00:39:17.140856 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.140798 5939 cpu_manager.go:414] "containerMap: removing container" podUID="266dfcf3-3908-4a9c-bb3c-c45b9407bc0c" containerName="s3-init" Oct 31 00:39:17.140856 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.140804 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="266dfcf3-3908-4a9c-bb3c-c45b9407bc0c" containerName="s3-init" Oct 31 00:39:17.140987 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.140869 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="266dfcf3-3908-4a9c-bb3c-c45b9407bc0c" containerName="s3-init" Oct 31 00:39:17.143963 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.143947 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-clr8b" Oct 31 00:39:17.151279 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.151255 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\"" Oct 31 00:39:17.151796 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.151779 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-f8d7d\"" Oct 31 00:39:17.154762 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.154744 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Oct 31 00:39:17.154852 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.154788 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Oct 31 00:39:17.155040 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.155022 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-clr8b"] Oct 31 00:39:17.237111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.237084 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tbsb\" (UniqueName: \"kubernetes.io/projected/95329c12-3826-4072-addd-3d6e33c1d9fb-kube-api-access-4tbsb\") pod \"seaweedfs-tls-custom-64cc6669cb-clr8b\" (UID: \"95329c12-3826-4072-addd-3d6e33c1d9fb\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-clr8b" Oct 31 00:39:17.237268 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.237132 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/95329c12-3826-4072-addd-3d6e33c1d9fb-data\") pod \"seaweedfs-tls-custom-64cc6669cb-clr8b\" (UID: \"95329c12-3826-4072-addd-3d6e33c1d9fb\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-clr8b" Oct 31 00:39:17.338035 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.338001 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tbsb\" (UniqueName: \"kubernetes.io/projected/95329c12-3826-4072-addd-3d6e33c1d9fb-kube-api-access-4tbsb\") pod \"seaweedfs-tls-custom-64cc6669cb-clr8b\" (UID: \"95329c12-3826-4072-addd-3d6e33c1d9fb\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-clr8b" Oct 31 00:39:17.338185 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.338046 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/95329c12-3826-4072-addd-3d6e33c1d9fb-data\") pod \"seaweedfs-tls-custom-64cc6669cb-clr8b\" (UID: \"95329c12-3826-4072-addd-3d6e33c1d9fb\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-clr8b" Oct 31 00:39:17.338356 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.338337 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/95329c12-3826-4072-addd-3d6e33c1d9fb-data\") pod \"seaweedfs-tls-custom-64cc6669cb-clr8b\" (UID: \"95329c12-3826-4072-addd-3d6e33c1d9fb\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-clr8b" Oct 31 00:39:17.352852 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.352823 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tbsb\" (UniqueName: \"kubernetes.io/projected/95329c12-3826-4072-addd-3d6e33c1d9fb-kube-api-access-4tbsb\") pod \"seaweedfs-tls-custom-64cc6669cb-clr8b\" (UID: \"95329c12-3826-4072-addd-3d6e33c1d9fb\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-clr8b" Oct 31 00:39:17.455125 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.455050 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-clr8b" Oct 31 00:39:17.731756 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.731685 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-clr8b"] Oct 31 00:39:17.734923 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:39:17.734897 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95329c12_3826_4072_addd_3d6e33c1d9fb.slice/crio-0d71036cb2e2fbb61a8e9af33fe5192223e34e9b0944049b06da48f363bcda05 WatchSource:0}: Error finding container 0d71036cb2e2fbb61a8e9af33fe5192223e34e9b0944049b06da48f363bcda05: Status 404 returned error can't find the container with id 0d71036cb2e2fbb61a8e9af33fe5192223e34e9b0944049b06da48f363bcda05 Oct 31 00:39:17.893909 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:17.893878 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-clr8b" event={"ID":"95329c12-3826-4072-addd-3d6e33c1d9fb","Type":"ContainerStarted","Data":"0d71036cb2e2fbb61a8e9af33fe5192223e34e9b0944049b06da48f363bcda05"} Oct 31 00:39:20.908196 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:20.908164 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-clr8b" event={"ID":"95329c12-3826-4072-addd-3d6e33c1d9fb","Type":"ContainerStarted","Data":"7ab676d5682a38dcc3102b2e5aa7917dc6d896ce8d41a0b788ba49b7ee685dc7"} Oct 31 00:39:20.930774 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:20.930727 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-64cc6669cb-clr8b" podStartSLOduration=1.581961113 podStartE2EDuration="3.930710947s" totalImagesPullingTime="2.348749834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:20.930019283 +0000 UTC m=+573.419294474" watchObservedRunningTime="2026-10-31 00:39:20.930710947 +0000 UTC m=+573.419986122" Oct 31 00:39:22.292799 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:22.292767 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-clr8b"] Oct 31 00:39:22.915312 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:22.915253 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-64cc6669cb-clr8b" podUID="95329c12-3826-4072-addd-3d6e33c1d9fb" containerName="seaweedfs-tls-custom" containerID="cri-o://7ab676d5682a38dcc3102b2e5aa7917dc6d896ce8d41a0b788ba49b7ee685dc7" gracePeriod=30 Oct 31 00:39:30.508450 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:39:30.508411 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:39:47.714401 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:47.714372 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:39:47.717266 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:47.717244 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:39:50.745627 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:50.745604 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-clr8b" Oct 31 00:39:50.816931 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:50.816899 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-685c9f986c-8lt65"] Oct 31 00:39:50.817273 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:50.817260 5939 cpu_manager.go:414] "containerMap: removing container" podUID="95329c12-3826-4072-addd-3d6e33c1d9fb" containerName="seaweedfs-tls-custom" Oct 31 00:39:50.817326 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:50.817275 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95329c12-3826-4072-addd-3d6e33c1d9fb" containerName="seaweedfs-tls-custom" Oct 31 00:39:50.817362 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:50.817337 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="95329c12-3826-4072-addd-3d6e33c1d9fb" containerName="seaweedfs-tls-custom" Oct 31 00:39:50.820760 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:50.820742 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-685c9f986c-8lt65" Oct 31 00:39:50.823349 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:50.823325 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95329c12-3826-4072-addd-3d6e33c1d9fb-kube-api-access-4tbsb\" (UniqueName: \"kubernetes.io/projected/95329c12-3826-4072-addd-3d6e33c1d9fb-kube-api-access-4tbsb\") pod \"95329c12-3826-4072-addd-3d6e33c1d9fb\" (UID: \"95329c12-3826-4072-addd-3d6e33c1d9fb\") " Oct 31 00:39:50.823509 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:50.823451 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95329c12-3826-4072-addd-3d6e33c1d9fb-data\" (UniqueName: \"kubernetes.io/empty-dir/95329c12-3826-4072-addd-3d6e33c1d9fb-data\") pod \"95329c12-3826-4072-addd-3d6e33c1d9fb\" (UID: \"95329c12-3826-4072-addd-3d6e33c1d9fb\") " Oct 31 00:39:50.824811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:50.824785 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95329c12-3826-4072-addd-3d6e33c1d9fb-data" pod "95329c12-3826-4072-addd-3d6e33c1d9fb" (UID: "95329c12-3826-4072-addd-3d6e33c1d9fb"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:39:50.825339 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:50.825318 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95329c12-3826-4072-addd-3d6e33c1d9fb-kube-api-access-4tbsb" pod "95329c12-3826-4072-addd-3d6e33c1d9fb" (UID: "95329c12-3826-4072-addd-3d6e33c1d9fb"). InnerVolumeSpecName "kube-api-access-4tbsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:50.826823 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:50.826803 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\"" Oct 31 00:39:50.831099 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:50.831080 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-685c9f986c-8lt65"] Oct 31 00:39:50.924657 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:50.924616 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gttr2\" (UniqueName: \"kubernetes.io/projected/23f1d2f3-affb-4340-8ed8-aa65b185f9da-kube-api-access-gttr2\") pod \"seaweedfs-tls-custom-685c9f986c-8lt65\" (UID: \"23f1d2f3-affb-4340-8ed8-aa65b185f9da\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-8lt65" Oct 31 00:39:50.924824 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:50.924689 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/23f1d2f3-affb-4340-8ed8-aa65b185f9da-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-685c9f986c-8lt65\" (UID: \"23f1d2f3-affb-4340-8ed8-aa65b185f9da\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-8lt65" Oct 31 00:39:50.924824 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:50.924804 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/23f1d2f3-affb-4340-8ed8-aa65b185f9da-data\") pod \"seaweedfs-tls-custom-685c9f986c-8lt65\" (UID: \"23f1d2f3-affb-4340-8ed8-aa65b185f9da\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-8lt65" Oct 31 00:39:50.924906 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:50.924851 5939 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/95329c12-3826-4072-addd-3d6e33c1d9fb-data\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:50.924906 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:50.924862 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tbsb\" (UniqueName: \"kubernetes.io/projected/95329c12-3826-4072-addd-3d6e33c1d9fb-kube-api-access-4tbsb\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:51.019285 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.019257 5939 generic.go:356] "Generic (PLEG): container finished" podID="95329c12-3826-4072-addd-3d6e33c1d9fb" containerID="7ab676d5682a38dcc3102b2e5aa7917dc6d896ce8d41a0b788ba49b7ee685dc7" exitCode=0 Oct 31 00:39:51.019457 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.019313 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-clr8b" Oct 31 00:39:51.019457 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.019343 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-clr8b" event={"ID":"95329c12-3826-4072-addd-3d6e33c1d9fb","Type":"ContainerDied","Data":"7ab676d5682a38dcc3102b2e5aa7917dc6d896ce8d41a0b788ba49b7ee685dc7"} Oct 31 00:39:51.019457 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.019422 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-clr8b" event={"ID":"95329c12-3826-4072-addd-3d6e33c1d9fb","Type":"ContainerDied","Data":"0d71036cb2e2fbb61a8e9af33fe5192223e34e9b0944049b06da48f363bcda05"} Oct 31 00:39:51.019457 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.019447 5939 scope.go:122] "RemoveContainer" containerID="7ab676d5682a38dcc3102b2e5aa7917dc6d896ce8d41a0b788ba49b7ee685dc7" Oct 31 00:39:51.025325 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.025305 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gttr2\" (UniqueName: \"kubernetes.io/projected/23f1d2f3-affb-4340-8ed8-aa65b185f9da-kube-api-access-gttr2\") pod \"seaweedfs-tls-custom-685c9f986c-8lt65\" (UID: \"23f1d2f3-affb-4340-8ed8-aa65b185f9da\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-8lt65" Oct 31 00:39:51.025428 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.025340 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/23f1d2f3-affb-4340-8ed8-aa65b185f9da-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-685c9f986c-8lt65\" (UID: \"23f1d2f3-affb-4340-8ed8-aa65b185f9da\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-8lt65" Oct 31 00:39:51.025428 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.025404 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/23f1d2f3-affb-4340-8ed8-aa65b185f9da-data\") pod \"seaweedfs-tls-custom-685c9f986c-8lt65\" (UID: \"23f1d2f3-affb-4340-8ed8-aa65b185f9da\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-8lt65" Oct 31 00:39:51.025757 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.025742 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/23f1d2f3-affb-4340-8ed8-aa65b185f9da-data\") pod \"seaweedfs-tls-custom-685c9f986c-8lt65\" (UID: \"23f1d2f3-affb-4340-8ed8-aa65b185f9da\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-8lt65" Oct 31 00:39:51.027967 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.027949 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/23f1d2f3-affb-4340-8ed8-aa65b185f9da-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-685c9f986c-8lt65\" (UID: \"23f1d2f3-affb-4340-8ed8-aa65b185f9da\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-8lt65" Oct 31 00:39:51.029552 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.029525 5939 scope.go:122] "RemoveContainer" containerID="7ab676d5682a38dcc3102b2e5aa7917dc6d896ce8d41a0b788ba49b7ee685dc7" Oct 31 00:39:51.029776 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:39:51.029754 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab676d5682a38dcc3102b2e5aa7917dc6d896ce8d41a0b788ba49b7ee685dc7\": container with ID starting with 7ab676d5682a38dcc3102b2e5aa7917dc6d896ce8d41a0b788ba49b7ee685dc7 not found: ID does not exist" containerID="7ab676d5682a38dcc3102b2e5aa7917dc6d896ce8d41a0b788ba49b7ee685dc7" Oct 31 00:39:51.029843 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.029788 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab676d5682a38dcc3102b2e5aa7917dc6d896ce8d41a0b788ba49b7ee685dc7"} err="failed to get container status \"7ab676d5682a38dcc3102b2e5aa7917dc6d896ce8d41a0b788ba49b7ee685dc7\": rpc error: code = NotFound desc = could not find container \"7ab676d5682a38dcc3102b2e5aa7917dc6d896ce8d41a0b788ba49b7ee685dc7\": container with ID starting with 7ab676d5682a38dcc3102b2e5aa7917dc6d896ce8d41a0b788ba49b7ee685dc7 not found: ID does not exist" Oct 31 00:39:51.040441 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.040420 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gttr2\" (UniqueName: \"kubernetes.io/projected/23f1d2f3-affb-4340-8ed8-aa65b185f9da-kube-api-access-gttr2\") pod \"seaweedfs-tls-custom-685c9f986c-8lt65\" (UID: \"23f1d2f3-affb-4340-8ed8-aa65b185f9da\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-8lt65" Oct 31 00:39:51.051624 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.051601 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-clr8b"] Oct 31 00:39:51.058658 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.058639 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-clr8b"] Oct 31 00:39:51.142058 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.142035 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-685c9f986c-8lt65" Oct 31 00:39:51.420235 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.420204 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-685c9f986c-8lt65"] Oct 31 00:39:51.423449 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:39:51.423421 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f1d2f3_affb_4340_8ed8_aa65b185f9da.slice/crio-f4f62216978bc33badd845fbb54c977549883d7585511f5f298de6d7bd32f5d7 WatchSource:0}: Error finding container f4f62216978bc33badd845fbb54c977549883d7585511f5f298de6d7bd32f5d7: Status 404 returned error can't find the container with id f4f62216978bc33badd845fbb54c977549883d7585511f5f298de6d7bd32f5d7 Oct 31 00:39:51.651929 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:51.651903 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95329c12-3826-4072-addd-3d6e33c1d9fb" path="/var/lib/kubelet/pods/95329c12-3826-4072-addd-3d6e33c1d9fb/volumes" Oct 31 00:39:52.024541 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:52.024452 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-685c9f986c-8lt65" event={"ID":"23f1d2f3-affb-4340-8ed8-aa65b185f9da","Type":"ContainerStarted","Data":"844108652fbd7b70cf384f46a592374f919fed3b3ccdfae5c7b1b950aa83deb5"} Oct 31 00:39:52.024541 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:52.024501 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-685c9f986c-8lt65" event={"ID":"23f1d2f3-affb-4340-8ed8-aa65b185f9da","Type":"ContainerStarted","Data":"f4f62216978bc33badd845fbb54c977549883d7585511f5f298de6d7bd32f5d7"} Oct 31 00:39:52.047678 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:52.047618 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-685c9f986c-8lt65" podStartSLOduration=1.716727987 podStartE2EDuration="2.047603094s" totalImagesPullingTime="330.875107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:52.046983094 +0000 UTC m=+604.536258272" watchObservedRunningTime="2026-10-31 00:39:52.047603094 +0000 UTC m=+604.536878251" Oct 31 00:39:52.564443 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:52.564410 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-25gmd"] Oct 31 00:39:52.568919 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:52.568902 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-25gmd" Oct 31 00:39:52.575394 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:52.575376 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\"" Oct 31 00:39:52.578895 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:52.578874 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-25gmd"] Oct 31 00:39:52.743192 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:52.743153 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sj5q\" (UniqueName: \"kubernetes.io/projected/8f54ff05-0753-4dfd-a703-c1743e79458e-kube-api-access-5sj5q\") pod \"s3-tls-init-custom-25gmd\" (UID: \"8f54ff05-0753-4dfd-a703-c1743e79458e\") " pod="kserve/s3-tls-init-custom-25gmd" Oct 31 00:39:52.743364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:52.743342 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f54ff05-0753-4dfd-a703-c1743e79458e-model-cache\") pod \"s3-tls-init-custom-25gmd\" (UID: \"8f54ff05-0753-4dfd-a703-c1743e79458e\") " pod="kserve/s3-tls-init-custom-25gmd" Oct 31 00:39:52.844080 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:52.844004 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f54ff05-0753-4dfd-a703-c1743e79458e-model-cache\") pod \"s3-tls-init-custom-25gmd\" (UID: \"8f54ff05-0753-4dfd-a703-c1743e79458e\") " pod="kserve/s3-tls-init-custom-25gmd" Oct 31 00:39:52.844080 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:52.844051 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sj5q\" (UniqueName: \"kubernetes.io/projected/8f54ff05-0753-4dfd-a703-c1743e79458e-kube-api-access-5sj5q\") pod \"s3-tls-init-custom-25gmd\" (UID: \"8f54ff05-0753-4dfd-a703-c1743e79458e\") " pod="kserve/s3-tls-init-custom-25gmd" Oct 31 00:39:52.844504 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:52.844480 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f54ff05-0753-4dfd-a703-c1743e79458e-model-cache\") pod \"s3-tls-init-custom-25gmd\" (UID: \"8f54ff05-0753-4dfd-a703-c1743e79458e\") " pod="kserve/s3-tls-init-custom-25gmd" Oct 31 00:39:52.859532 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:52.859500 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sj5q\" (UniqueName: \"kubernetes.io/projected/8f54ff05-0753-4dfd-a703-c1743e79458e-kube-api-access-5sj5q\") pod \"s3-tls-init-custom-25gmd\" (UID: \"8f54ff05-0753-4dfd-a703-c1743e79458e\") " pod="kserve/s3-tls-init-custom-25gmd" Oct 31 00:39:52.891272 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:52.891244 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-25gmd" Oct 31 00:39:53.169499 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:53.169467 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-25gmd"] Oct 31 00:39:53.173230 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:39:53.173204 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f54ff05_0753_4dfd_a703_c1743e79458e.slice/crio-e1c09dcab80ca625445caea19242a52bda905ab041486f238aadeaf750a9085b WatchSource:0}: Error finding container e1c09dcab80ca625445caea19242a52bda905ab041486f238aadeaf750a9085b: Status 404 returned error can't find the container with id e1c09dcab80ca625445caea19242a52bda905ab041486f238aadeaf750a9085b Oct 31 00:39:54.034848 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:54.034810 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-25gmd" event={"ID":"8f54ff05-0753-4dfd-a703-c1743e79458e","Type":"ContainerStarted","Data":"a03b744ea83dd7c000c88a66b27afb0b1a9b48cdc5b939cd9da9f1c9aafe7902"} Oct 31 00:39:54.034848 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:54.034856 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-25gmd" event={"ID":"8f54ff05-0753-4dfd-a703-c1743e79458e","Type":"ContainerStarted","Data":"e1c09dcab80ca625445caea19242a52bda905ab041486f238aadeaf750a9085b"} Oct 31 00:39:55.039976 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:55.039885 5939 generic.go:356] "Generic (PLEG): container finished" podID="8f54ff05-0753-4dfd-a703-c1743e79458e" containerID="a03b744ea83dd7c000c88a66b27afb0b1a9b48cdc5b939cd9da9f1c9aafe7902" exitCode=0 Oct 31 00:39:55.039976 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:55.039955 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-25gmd" event={"ID":"8f54ff05-0753-4dfd-a703-c1743e79458e","Type":"ContainerDied","Data":"a03b744ea83dd7c000c88a66b27afb0b1a9b48cdc5b939cd9da9f1c9aafe7902"} Oct 31 00:39:56.045907 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:56.045865 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-25gmd" event={"ID":"8f54ff05-0753-4dfd-a703-c1743e79458e","Type":"ContainerStarted","Data":"3512f05fc587e17061e67589fa2dd3b50ce738859fb73179b911299d88d9bea4"} Oct 31 00:39:56.068992 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:39:56.068946 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-25gmd" podStartSLOduration=2.666869039 podStartE2EDuration="4.068934045s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.402065006s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:56.068628678 +0000 UTC m=+608.557903857" watchObservedRunningTime="2026-10-31 00:39:56.068934045 +0000 UTC m=+608.558209224" Oct 31 00:40:00.509193 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:40:00.509147 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:40:02.073360 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:02.073331 5939 generic.go:356] "Generic (PLEG): container finished" podID="8f54ff05-0753-4dfd-a703-c1743e79458e" containerID="3512f05fc587e17061e67589fa2dd3b50ce738859fb73179b911299d88d9bea4" exitCode=0 Oct 31 00:40:02.073751 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:02.073379 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-25gmd" event={"ID":"8f54ff05-0753-4dfd-a703-c1743e79458e","Type":"ContainerDied","Data":"3512f05fc587e17061e67589fa2dd3b50ce738859fb73179b911299d88d9bea4"} Oct 31 00:40:03.204627 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:03.204601 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-25gmd" Oct 31 00:40:03.241962 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:03.241933 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f54ff05-0753-4dfd-a703-c1743e79458e-kube-api-access-5sj5q\" (UniqueName: \"kubernetes.io/projected/8f54ff05-0753-4dfd-a703-c1743e79458e-kube-api-access-5sj5q\") pod \"8f54ff05-0753-4dfd-a703-c1743e79458e\" (UID: \"8f54ff05-0753-4dfd-a703-c1743e79458e\") " Oct 31 00:40:03.242062 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:03.242025 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f54ff05-0753-4dfd-a703-c1743e79458e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f54ff05-0753-4dfd-a703-c1743e79458e-model-cache\") pod \"8f54ff05-0753-4dfd-a703-c1743e79458e\" (UID: \"8f54ff05-0753-4dfd-a703-c1743e79458e\") " Oct 31 00:40:03.243875 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:03.243850 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f54ff05-0753-4dfd-a703-c1743e79458e-kube-api-access-5sj5q" pod "8f54ff05-0753-4dfd-a703-c1743e79458e" (UID: "8f54ff05-0753-4dfd-a703-c1743e79458e"). InnerVolumeSpecName "kube-api-access-5sj5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:03.262994 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:03.262971 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f54ff05-0753-4dfd-a703-c1743e79458e-model-cache" pod "8f54ff05-0753-4dfd-a703-c1743e79458e" (UID: "8f54ff05-0753-4dfd-a703-c1743e79458e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:03.342893 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:03.342843 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sj5q\" (UniqueName: \"kubernetes.io/projected/8f54ff05-0753-4dfd-a703-c1743e79458e-kube-api-access-5sj5q\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:03.342893 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:03.342864 5939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f54ff05-0753-4dfd-a703-c1743e79458e-model-cache\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:04.083154 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:04.083080 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-25gmd" Oct 31 00:40:04.083294 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:04.083082 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-25gmd" event={"ID":"8f54ff05-0753-4dfd-a703-c1743e79458e","Type":"ContainerDied","Data":"e1c09dcab80ca625445caea19242a52bda905ab041486f238aadeaf750a9085b"} Oct 31 00:40:04.083294 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:04.083192 5939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1c09dcab80ca625445caea19242a52bda905ab041486f238aadeaf750a9085b" Oct 31 00:40:05.148832 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.148805 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-7qnl4"] Oct 31 00:40:05.149211 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.149116 5939 cpu_manager.go:414] "containerMap: removing container" podUID="8f54ff05-0753-4dfd-a703-c1743e79458e" containerName="s3-tls-init-custom" Oct 31 00:40:05.149211 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.149126 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f54ff05-0753-4dfd-a703-c1743e79458e" containerName="s3-tls-init-custom" Oct 31 00:40:05.149211 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.149160 5939 cpu_manager.go:414] "containerMap: removing container" podUID="8f54ff05-0753-4dfd-a703-c1743e79458e" containerName="download-hf-model" Oct 31 00:40:05.149211 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.149165 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f54ff05-0753-4dfd-a703-c1743e79458e" containerName="download-hf-model" Oct 31 00:40:05.149211 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.149210 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f54ff05-0753-4dfd-a703-c1743e79458e" containerName="s3-tls-init-custom" Oct 31 00:40:05.153155 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.153134 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-5c74974785-7qnl4" Oct 31 00:40:05.153453 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.153429 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/0489b906-3432-46a6-9c0b-2ef27f65d8fb-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-7qnl4\" (UID: \"0489b906-3432-46a6-9c0b-2ef27f65d8fb\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7qnl4" Oct 31 00:40:05.153580 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.153557 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tklh\" (UniqueName: \"kubernetes.io/projected/0489b906-3432-46a6-9c0b-2ef27f65d8fb-kube-api-access-6tklh\") pod \"seaweedfs-tls-serving-5c74974785-7qnl4\" (UID: \"0489b906-3432-46a6-9c0b-2ef27f65d8fb\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7qnl4" Oct 31 00:40:05.153721 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.153699 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0489b906-3432-46a6-9c0b-2ef27f65d8fb-data\") pod \"seaweedfs-tls-serving-5c74974785-7qnl4\" (UID: \"0489b906-3432-46a6-9c0b-2ef27f65d8fb\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7qnl4" Oct 31 00:40:05.160012 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.159999 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\"" Oct 31 00:40:05.160190 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.160178 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\"" Oct 31 00:40:05.163652 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.163633 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-7qnl4"] Oct 31 00:40:05.254708 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.254661 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tklh\" (UniqueName: \"kubernetes.io/projected/0489b906-3432-46a6-9c0b-2ef27f65d8fb-kube-api-access-6tklh\") pod \"seaweedfs-tls-serving-5c74974785-7qnl4\" (UID: \"0489b906-3432-46a6-9c0b-2ef27f65d8fb\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7qnl4" Oct 31 00:40:05.254827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.254755 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0489b906-3432-46a6-9c0b-2ef27f65d8fb-data\") pod \"seaweedfs-tls-serving-5c74974785-7qnl4\" (UID: \"0489b906-3432-46a6-9c0b-2ef27f65d8fb\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7qnl4" Oct 31 00:40:05.254827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.254790 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/0489b906-3432-46a6-9c0b-2ef27f65d8fb-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-7qnl4\" (UID: \"0489b906-3432-46a6-9c0b-2ef27f65d8fb\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7qnl4" Oct 31 00:40:05.255165 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.255146 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0489b906-3432-46a6-9c0b-2ef27f65d8fb-data\") pod \"seaweedfs-tls-serving-5c74974785-7qnl4\" (UID: \"0489b906-3432-46a6-9c0b-2ef27f65d8fb\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7qnl4" Oct 31 00:40:05.257153 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.257128 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/0489b906-3432-46a6-9c0b-2ef27f65d8fb-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-7qnl4\" (UID: \"0489b906-3432-46a6-9c0b-2ef27f65d8fb\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7qnl4" Oct 31 00:40:05.269386 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.269364 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tklh\" (UniqueName: \"kubernetes.io/projected/0489b906-3432-46a6-9c0b-2ef27f65d8fb-kube-api-access-6tklh\") pod \"seaweedfs-tls-serving-5c74974785-7qnl4\" (UID: \"0489b906-3432-46a6-9c0b-2ef27f65d8fb\") " pod="kserve/seaweedfs-tls-serving-5c74974785-7qnl4" Oct 31 00:40:05.465425 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.465341 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-5c74974785-7qnl4" Oct 31 00:40:05.742739 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:05.742647 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-7qnl4"] Oct 31 00:40:05.745980 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:40:05.745954 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0489b906_3432_46a6_9c0b_2ef27f65d8fb.slice/crio-cc6318a142463ec945fbca0c2c645dd581ba1b20cfd024a13312d99791049730 WatchSource:0}: Error finding container cc6318a142463ec945fbca0c2c645dd581ba1b20cfd024a13312d99791049730: Status 404 returned error can't find the container with id cc6318a142463ec945fbca0c2c645dd581ba1b20cfd024a13312d99791049730 Oct 31 00:40:06.091869 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:06.091796 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-5c74974785-7qnl4" event={"ID":"0489b906-3432-46a6-9c0b-2ef27f65d8fb","Type":"ContainerStarted","Data":"cc6318a142463ec945fbca0c2c645dd581ba1b20cfd024a13312d99791049730"} Oct 31 00:40:07.097505 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:07.097463 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-5c74974785-7qnl4" event={"ID":"0489b906-3432-46a6-9c0b-2ef27f65d8fb","Type":"ContainerStarted","Data":"13820c6fe9fa3ff16d95e2c72b76416d42f28d0a0433215f8e9c2a901da104d3"} Oct 31 00:40:07.983982 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:07.983937 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-5c74974785-7qnl4" podStartSLOduration=2.60999091 podStartE2EDuration="2.983922886s" totalImagesPullingTime="373.931976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:07.12163328 +0000 UTC m=+619.610908457" watchObservedRunningTime="2026-10-31 00:40:07.983922886 +0000 UTC m=+620.473198066" Oct 31 00:40:07.984150 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:07.984139 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-9qhsg"] Oct 31 00:40:07.988612 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:07.988593 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9qhsg" Oct 31 00:40:07.995201 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:07.994976 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\"" Oct 31 00:40:07.998912 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:07.998890 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-9qhsg"] Oct 31 00:40:08.076782 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:08.076747 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97cda6ca-3db4-46be-ba8f-fb77be8db868-model-cache\") pod \"s3-tls-init-serving-9qhsg\" (UID: \"97cda6ca-3db4-46be-ba8f-fb77be8db868\") " pod="kserve/s3-tls-init-serving-9qhsg" Oct 31 00:40:08.076919 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:08.076810 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7w5h\" (UniqueName: \"kubernetes.io/projected/97cda6ca-3db4-46be-ba8f-fb77be8db868-kube-api-access-q7w5h\") pod \"s3-tls-init-serving-9qhsg\" (UID: \"97cda6ca-3db4-46be-ba8f-fb77be8db868\") " pod="kserve/s3-tls-init-serving-9qhsg" Oct 31 00:40:08.177702 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:08.177654 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7w5h\" (UniqueName: \"kubernetes.io/projected/97cda6ca-3db4-46be-ba8f-fb77be8db868-kube-api-access-q7w5h\") pod \"s3-tls-init-serving-9qhsg\" (UID: \"97cda6ca-3db4-46be-ba8f-fb77be8db868\") " pod="kserve/s3-tls-init-serving-9qhsg" Oct 31 00:40:08.178120 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:08.177876 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97cda6ca-3db4-46be-ba8f-fb77be8db868-model-cache\") pod \"s3-tls-init-serving-9qhsg\" (UID: \"97cda6ca-3db4-46be-ba8f-fb77be8db868\") " pod="kserve/s3-tls-init-serving-9qhsg" Oct 31 00:40:08.178250 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:08.178230 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97cda6ca-3db4-46be-ba8f-fb77be8db868-model-cache\") pod \"s3-tls-init-serving-9qhsg\" (UID: \"97cda6ca-3db4-46be-ba8f-fb77be8db868\") " pod="kserve/s3-tls-init-serving-9qhsg" Oct 31 00:40:08.192770 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:08.192748 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7w5h\" (UniqueName: \"kubernetes.io/projected/97cda6ca-3db4-46be-ba8f-fb77be8db868-kube-api-access-q7w5h\") pod \"s3-tls-init-serving-9qhsg\" (UID: \"97cda6ca-3db4-46be-ba8f-fb77be8db868\") " pod="kserve/s3-tls-init-serving-9qhsg" Oct 31 00:40:08.309944 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:08.309879 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9qhsg" Oct 31 00:40:08.584628 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:08.584555 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-9qhsg"] Oct 31 00:40:08.588007 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:40:08.587976 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97cda6ca_3db4_46be_ba8f_fb77be8db868.slice/crio-e37e61c3c93b6e2acb52f4ebf3d11a716b6d6de8559b505bc854023bd5345154 WatchSource:0}: Error finding container e37e61c3c93b6e2acb52f4ebf3d11a716b6d6de8559b505bc854023bd5345154: Status 404 returned error can't find the container with id e37e61c3c93b6e2acb52f4ebf3d11a716b6d6de8559b505bc854023bd5345154 Oct 31 00:40:09.106821 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:09.106785 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9qhsg" event={"ID":"97cda6ca-3db4-46be-ba8f-fb77be8db868","Type":"ContainerStarted","Data":"474e9f23b1701d26367363c11735141ab6c87c61e53503069d55fb9d9ef2455f"} Oct 31 00:40:09.106821 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:09.106830 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9qhsg" event={"ID":"97cda6ca-3db4-46be-ba8f-fb77be8db868","Type":"ContainerStarted","Data":"e37e61c3c93b6e2acb52f4ebf3d11a716b6d6de8559b505bc854023bd5345154"} Oct 31 00:40:10.112110 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:10.112076 5939 generic.go:356] "Generic (PLEG): container finished" podID="97cda6ca-3db4-46be-ba8f-fb77be8db868" containerID="474e9f23b1701d26367363c11735141ab6c87c61e53503069d55fb9d9ef2455f" exitCode=0 Oct 31 00:40:10.112480 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:10.112155 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9qhsg" event={"ID":"97cda6ca-3db4-46be-ba8f-fb77be8db868","Type":"ContainerDied","Data":"474e9f23b1701d26367363c11735141ab6c87c61e53503069d55fb9d9ef2455f"} Oct 31 00:40:11.117520 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:11.117479 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9qhsg" event={"ID":"97cda6ca-3db4-46be-ba8f-fb77be8db868","Type":"ContainerStarted","Data":"29e72a1575e30b01ca9afdf66dc827086d9b811a0e7340617eee601526c0058a"} Oct 31 00:40:11.141129 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:11.141084 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-9qhsg" podStartSLOduration=2.744367669 podStartE2EDuration="4.141071372s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.396703703s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:11.140425961 +0000 UTC m=+623.629701153" watchObservedRunningTime="2026-10-31 00:40:11.141071372 +0000 UTC m=+623.630346580" Oct 31 00:40:16.140886 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:16.140858 5939 generic.go:356] "Generic (PLEG): container finished" podID="97cda6ca-3db4-46be-ba8f-fb77be8db868" containerID="29e72a1575e30b01ca9afdf66dc827086d9b811a0e7340617eee601526c0058a" exitCode=0 Oct 31 00:40:16.141254 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:16.140892 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9qhsg" event={"ID":"97cda6ca-3db4-46be-ba8f-fb77be8db868","Type":"ContainerDied","Data":"29e72a1575e30b01ca9afdf66dc827086d9b811a0e7340617eee601526c0058a"} Oct 31 00:40:17.282450 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:17.282419 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9qhsg" Oct 31 00:40:17.365433 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:17.365396 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97cda6ca-3db4-46be-ba8f-fb77be8db868-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97cda6ca-3db4-46be-ba8f-fb77be8db868-model-cache\") pod \"97cda6ca-3db4-46be-ba8f-fb77be8db868\" (UID: \"97cda6ca-3db4-46be-ba8f-fb77be8db868\") " Oct 31 00:40:17.365605 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:17.365468 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97cda6ca-3db4-46be-ba8f-fb77be8db868-kube-api-access-q7w5h\" (UniqueName: \"kubernetes.io/projected/97cda6ca-3db4-46be-ba8f-fb77be8db868-kube-api-access-q7w5h\") pod \"97cda6ca-3db4-46be-ba8f-fb77be8db868\" (UID: \"97cda6ca-3db4-46be-ba8f-fb77be8db868\") " Oct 31 00:40:17.367515 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:17.367481 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97cda6ca-3db4-46be-ba8f-fb77be8db868-kube-api-access-q7w5h" pod "97cda6ca-3db4-46be-ba8f-fb77be8db868" (UID: "97cda6ca-3db4-46be-ba8f-fb77be8db868"). InnerVolumeSpecName "kube-api-access-q7w5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:17.387006 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:17.386977 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97cda6ca-3db4-46be-ba8f-fb77be8db868-model-cache" pod "97cda6ca-3db4-46be-ba8f-fb77be8db868" (UID: "97cda6ca-3db4-46be-ba8f-fb77be8db868"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:17.467093 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:17.467030 5939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97cda6ca-3db4-46be-ba8f-fb77be8db868-model-cache\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:17.467093 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:17.467055 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7w5h\" (UniqueName: \"kubernetes.io/projected/97cda6ca-3db4-46be-ba8f-fb77be8db868-kube-api-access-q7w5h\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:18.149430 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:18.149390 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9qhsg" event={"ID":"97cda6ca-3db4-46be-ba8f-fb77be8db868","Type":"ContainerDied","Data":"e37e61c3c93b6e2acb52f4ebf3d11a716b6d6de8559b505bc854023bd5345154"} Oct 31 00:40:18.149430 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:18.149428 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9qhsg" Oct 31 00:40:18.149430 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:18.149435 5939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e37e61c3c93b6e2acb52f4ebf3d11a716b6d6de8559b505bc854023bd5345154" Oct 31 00:40:30.508468 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:40:30.508424 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:40:39.807124 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.807085 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp"] Oct 31 00:40:39.807756 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.807622 5939 cpu_manager.go:414] "containerMap: removing container" podUID="97cda6ca-3db4-46be-ba8f-fb77be8db868" containerName="s3-tls-init-serving" Oct 31 00:40:39.807756 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.807642 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97cda6ca-3db4-46be-ba8f-fb77be8db868" containerName="s3-tls-init-serving" Oct 31 00:40:39.807756 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.807660 5939 cpu_manager.go:414] "containerMap: removing container" podUID="97cda6ca-3db4-46be-ba8f-fb77be8db868" containerName="download-hf-model" Oct 31 00:40:39.807756 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.807686 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97cda6ca-3db4-46be-ba8f-fb77be8db868" containerName="download-hf-model" Oct 31 00:40:39.807982 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.807764 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="97cda6ca-3db4-46be-ba8f-fb77be8db868" containerName="s3-tls-init-serving" Oct 31 00:40:39.811159 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.811137 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:39.817327 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.817303 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:40:39.817438 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.817352 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\"" Oct 31 00:40:39.818050 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.818003 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 00:40:39.818175 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.818066 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 00:40:39.818175 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.818068 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 00:40:39.818175 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.818078 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\"" Oct 31 00:40:39.818175 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.818139 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 00:40:39.826136 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.826115 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp"] Oct 31 00:40:39.959845 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.959814 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a453380-7b94-406a-a376-489b0709d0bd-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:39.959998 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.959877 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a453380-7b94-406a-a376-489b0709d0bd-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:39.959998 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.959954 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a453380-7b94-406a-a376-489b0709d0bd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:39.959998 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.959989 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a453380-7b94-406a-a376-489b0709d0bd-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:39.960095 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:39.960017 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmd2d\" (UniqueName: \"kubernetes.io/projected/5a453380-7b94-406a-a376-489b0709d0bd-kube-api-access-tmd2d\") pod \"isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:40.061344 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.061275 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmd2d\" (UniqueName: \"kubernetes.io/projected/5a453380-7b94-406a-a376-489b0709d0bd-kube-api-access-tmd2d\") pod \"isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:40.061344 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.061318 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a453380-7b94-406a-a376-489b0709d0bd-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:40.061574 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.061367 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a453380-7b94-406a-a376-489b0709d0bd-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:40.061574 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.061387 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a453380-7b94-406a-a376-489b0709d0bd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:40.061574 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.061412 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a453380-7b94-406a-a376-489b0709d0bd-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:40.061856 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.061836 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a453380-7b94-406a-a376-489b0709d0bd-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:40.062138 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.062115 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a453380-7b94-406a-a376-489b0709d0bd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:40.062138 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.062129 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a453380-7b94-406a-a376-489b0709d0bd-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:40.063817 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.063799 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a453380-7b94-406a-a376-489b0709d0bd-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:40.076131 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.076112 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmd2d\" (UniqueName: \"kubernetes.io/projected/5a453380-7b94-406a-a376-489b0709d0bd-kube-api-access-tmd2d\") pod \"isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:40.084538 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.084516 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw"] Oct 31 00:40:40.088572 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.088557 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:40:40.095153 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.094956 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\"" Oct 31 00:40:40.095153 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.094956 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\"" Oct 31 00:40:40.095153 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.095006 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:40:40.095153 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.094961 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\"" Oct 31 00:40:40.095153 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.095019 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-sklearn-batcher-predictor-serving-cert\"" Oct 31 00:40:40.095153 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.095068 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\"" Oct 31 00:40:40.095153 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.095077 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\"" Oct 31 00:40:40.103499 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.103480 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw"] Oct 31 00:40:40.127920 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.127897 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:40.162054 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.162026 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa8cd3f-efe6-4524-a192-6888d26e6a05-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7555846756-ntwzw\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:40:40.162194 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.162070 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwzbk\" (UniqueName: \"kubernetes.io/projected/daa8cd3f-efe6-4524-a192-6888d26e6a05-kube-api-access-kwzbk\") pod \"isvc-sklearn-batcher-predictor-7555846756-ntwzw\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:40:40.162194 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.162099 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa8cd3f-efe6-4524-a192-6888d26e6a05-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7555846756-ntwzw\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:40:40.162277 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.162197 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa8cd3f-efe6-4524-a192-6888d26e6a05-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7555846756-ntwzw\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:40:40.162277 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.162254 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa8cd3f-efe6-4524-a192-6888d26e6a05-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7555846756-ntwzw\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:40:40.263350 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.263318 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa8cd3f-efe6-4524-a192-6888d26e6a05-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7555846756-ntwzw\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:40:40.263508 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.263364 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwzbk\" (UniqueName: \"kubernetes.io/projected/daa8cd3f-efe6-4524-a192-6888d26e6a05-kube-api-access-kwzbk\") pod \"isvc-sklearn-batcher-predictor-7555846756-ntwzw\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:40:40.263508 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.263390 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa8cd3f-efe6-4524-a192-6888d26e6a05-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7555846756-ntwzw\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:40:40.263508 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.263421 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa8cd3f-efe6-4524-a192-6888d26e6a05-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7555846756-ntwzw\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:40:40.263508 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.263459 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa8cd3f-efe6-4524-a192-6888d26e6a05-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7555846756-ntwzw\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:40:40.263820 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.263798 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa8cd3f-efe6-4524-a192-6888d26e6a05-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7555846756-ntwzw\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:40:40.264062 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.264032 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa8cd3f-efe6-4524-a192-6888d26e6a05-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7555846756-ntwzw\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:40:40.264163 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.264100 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa8cd3f-efe6-4524-a192-6888d26e6a05-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7555846756-ntwzw\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:40:40.265975 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.265956 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa8cd3f-efe6-4524-a192-6888d26e6a05-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7555846756-ntwzw\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:40:40.278447 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.278423 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwzbk\" (UniqueName: \"kubernetes.io/projected/daa8cd3f-efe6-4524-a192-6888d26e6a05-kube-api-access-kwzbk\") pod \"isvc-sklearn-batcher-predictor-7555846756-ntwzw\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:40:40.407066 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.406895 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp"] Oct 31 00:40:40.408763 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.408737 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:40:40.409747 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:40:40.409719 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a453380_7b94_406a_a376_489b0709d0bd.slice/crio-82d6e85f64b936e9377beaf8e0f638ad8ca37cde11b8c30b3fa681e89c3eccb3 WatchSource:0}: Error finding container 82d6e85f64b936e9377beaf8e0f638ad8ca37cde11b8c30b3fa681e89c3eccb3: Status 404 returned error can't find the container with id 82d6e85f64b936e9377beaf8e0f638ad8ca37cde11b8c30b3fa681e89c3eccb3 Oct 31 00:40:40.695419 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:40.695343 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw"] Oct 31 00:40:40.698474 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:40:40.698446 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa8cd3f_efe6_4524_a192_6888d26e6a05.slice/crio-b41af9749e8f86f46fdcddbdd46104dbcb3bba5d4eb2cd40489334fc7b2a2c9e WatchSource:0}: Error finding container b41af9749e8f86f46fdcddbdd46104dbcb3bba5d4eb2cd40489334fc7b2a2c9e: Status 404 returned error can't find the container with id b41af9749e8f86f46fdcddbdd46104dbcb3bba5d4eb2cd40489334fc7b2a2c9e Oct 31 00:40:41.245271 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:41.245230 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" event={"ID":"daa8cd3f-efe6-4524-a192-6888d26e6a05","Type":"ContainerStarted","Data":"b9b3bb4da830bc8de2ed61378fe9b672b6dce73462f1bda74398359c92e40e3b"} Oct 31 00:40:41.245716 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:41.245343 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" event={"ID":"daa8cd3f-efe6-4524-a192-6888d26e6a05","Type":"ContainerStarted","Data":"b41af9749e8f86f46fdcddbdd46104dbcb3bba5d4eb2cd40489334fc7b2a2c9e"} Oct 31 00:40:41.246922 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:41.246896 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" event={"ID":"5a453380-7b94-406a-a376-489b0709d0bd","Type":"ContainerStarted","Data":"7e272f057ee94296ec3f8e6246fe408ce4edf9f5f25aae7a89e9344d80d4d636"} Oct 31 00:40:41.247047 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:41.246955 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" event={"ID":"5a453380-7b94-406a-a376-489b0709d0bd","Type":"ContainerStarted","Data":"82d6e85f64b936e9377beaf8e0f638ad8ca37cde11b8c30b3fa681e89c3eccb3"} Oct 31 00:40:45.265863 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:45.265784 5939 generic.go:356] "Generic (PLEG): container finished" podID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerID="b9b3bb4da830bc8de2ed61378fe9b672b6dce73462f1bda74398359c92e40e3b" exitCode=0 Oct 31 00:40:45.265863 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:45.265853 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" event={"ID":"daa8cd3f-efe6-4524-a192-6888d26e6a05","Type":"ContainerDied","Data":"b9b3bb4da830bc8de2ed61378fe9b672b6dce73462f1bda74398359c92e40e3b"} Oct 31 00:40:45.267783 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:45.267629 5939 generic.go:356] "Generic (PLEG): container finished" podID="5a453380-7b94-406a-a376-489b0709d0bd" containerID="7e272f057ee94296ec3f8e6246fe408ce4edf9f5f25aae7a89e9344d80d4d636" exitCode=0 Oct 31 00:40:45.267783 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:45.267721 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" event={"ID":"5a453380-7b94-406a-a376-489b0709d0bd","Type":"ContainerDied","Data":"7e272f057ee94296ec3f8e6246fe408ce4edf9f5f25aae7a89e9344d80d4d636"} Oct 31 00:40:56.368384 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:56.368345 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" event={"ID":"5a453380-7b94-406a-a376-489b0709d0bd","Type":"ContainerStarted","Data":"73a4b5b28aee8cdca46acd83bc18a72c12279e6e3a22a4cc422cb438c1ed08ac"} Oct 31 00:40:59.395248 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:59.395211 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" event={"ID":"5a453380-7b94-406a-a376-489b0709d0bd","Type":"ContainerStarted","Data":"c2457ecba97a133d5d6f55f42c7387d3e288e8680d310a89fa83725e8fac8ae1"} Oct 31 00:40:59.395722 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:59.395338 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:40:59.424678 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:40:59.424604 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" podStartSLOduration=2.533760193 podStartE2EDuration="20.424589444s" totalImagesPullingTime="13.434952574s" totalInitContainerRuntime="4.455876677s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:59.423093176 +0000 UTC m=+671.912368366" watchObservedRunningTime="2026-10-31 00:40:59.424589444 +0000 UTC m=+671.913864624" Oct 31 00:41:00.400622 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:00.400586 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:41:00.402132 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:00.402097 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 31 00:41:00.511158 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:41:00.511119 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:41:01.405851 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:01.405807 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 31 00:41:03.420376 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:03.420344 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" event={"ID":"daa8cd3f-efe6-4524-a192-6888d26e6a05","Type":"ContainerStarted","Data":"46ea9c0af6a1d97bb4bc56ddf90cc7ae85effa6a4f25ef95b63e46df44e2765c"} Oct 31 00:41:03.420770 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:03.420422 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" event={"ID":"daa8cd3f-efe6-4524-a192-6888d26e6a05","Type":"ContainerStarted","Data":"d8fce6ac31aec5ff2f68ba48c667f71e205b7e00aa75a38e9978015d4307abe9"} Oct 31 00:41:06.410215 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:06.410189 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:41:06.410690 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:06.410649 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 31 00:41:06.440234 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:06.440202 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" event={"ID":"daa8cd3f-efe6-4524-a192-6888d26e6a05","Type":"ContainerStarted","Data":"512c9a739830ad3ebe76af44c29ef80b01c51617800dd5a6ad9db14e42c7e953"} Oct 31 00:41:06.440414 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:06.440380 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:41:06.473240 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:06.472999 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podStartSLOduration=1.963561967 podStartE2EDuration="26.47298118s" totalImagesPullingTime="20.628011336s" totalInitContainerRuntime="3.881407877s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:06.471874634 +0000 UTC m=+678.961149838" watchObservedRunningTime="2026-10-31 00:41:06.47298118 +0000 UTC m=+678.962256360" Oct 31 00:41:07.444522 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:07.444491 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:41:07.444954 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:07.444561 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:41:07.445804 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:07.445774 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 31 00:41:07.446478 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:07.446456 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:41:08.449140 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:08.449081 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 31 00:41:08.449634 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:08.449476 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:41:08.453445 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:08.453422 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:41:09.457110 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:09.457077 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 31 00:41:09.457591 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:09.457567 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:41:16.410990 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:16.410949 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 31 00:41:19.457807 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:19.457762 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 31 00:41:19.458202 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:19.458176 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:41:26.411294 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:26.411253 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 31 00:41:29.457231 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:29.457176 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 31 00:41:29.457786 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:29.457759 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:41:30.509775 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:41:30.509740 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:41:36.411268 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:36.411224 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 31 00:41:39.457090 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:39.457043 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 31 00:41:39.457641 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:39.457516 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:41:46.410867 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:46.410828 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 31 00:41:49.457984 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:49.457926 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 31 00:41:49.458369 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:49.458344 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:41:56.410837 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:56.410797 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 31 00:41:59.457547 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:59.457444 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 31 00:41:59.457994 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:41:59.457904 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:42:00.508608 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:42:00.508570 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:42:06.410723 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:06.410686 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 31 00:42:09.457116 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:09.457069 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 31 00:42:09.457620 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:09.457599 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:42:11.652474 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:11.652428 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 31 00:42:19.457339 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:19.457286 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 31 00:42:19.457863 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:19.457787 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:42:21.653281 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:21.653239 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 31 00:42:22.647844 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:22.647813 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:42:22.648009 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:22.647950 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:42:30.508884 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:42:30.508846 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:42:31.652816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:31.652788 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:42:35.669534 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:35.669505 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw"] Oct 31 00:42:35.669983 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:35.669747 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kserve-container" containerID="cri-o://d8fce6ac31aec5ff2f68ba48c667f71e205b7e00aa75a38e9978015d4307abe9" gracePeriod=30 Oct 31 00:42:35.669983 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:35.669763 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="agent" containerID="cri-o://512c9a739830ad3ebe76af44c29ef80b01c51617800dd5a6ad9db14e42c7e953" gracePeriod=30 Oct 31 00:42:35.669983 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:35.669813 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kube-rbac-proxy" containerID="cri-o://46ea9c0af6a1d97bb4bc56ddf90cc7ae85effa6a4f25ef95b63e46df44e2765c" gracePeriod=30 Oct 31 00:42:35.832104 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:35.832077 5939 generic.go:356] "Generic (PLEG): container finished" podID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerID="46ea9c0af6a1d97bb4bc56ddf90cc7ae85effa6a4f25ef95b63e46df44e2765c" exitCode=0 Oct 31 00:42:35.832244 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:35.832139 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" event={"ID":"daa8cd3f-efe6-4524-a192-6888d26e6a05","Type":"ContainerDied","Data":"46ea9c0af6a1d97bb4bc56ddf90cc7ae85effa6a4f25ef95b63e46df44e2765c"} Oct 31 00:42:38.449901 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:38.449862 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused" Oct 31 00:42:40.854816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:40.854782 5939 generic.go:356] "Generic (PLEG): container finished" podID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerID="d8fce6ac31aec5ff2f68ba48c667f71e205b7e00aa75a38e9978015d4307abe9" exitCode=0 Oct 31 00:42:40.855231 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:40.854849 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" event={"ID":"daa8cd3f-efe6-4524-a192-6888d26e6a05","Type":"ContainerDied","Data":"d8fce6ac31aec5ff2f68ba48c667f71e205b7e00aa75a38e9978015d4307abe9"} Oct 31 00:42:41.614193 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:41.614163 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp"] Oct 31 00:42:41.614401 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:41.614381 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kserve-container" containerID="cri-o://73a4b5b28aee8cdca46acd83bc18a72c12279e6e3a22a4cc422cb438c1ed08ac" gracePeriod=30 Oct 31 00:42:41.614474 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:41.614435 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kube-rbac-proxy" containerID="cri-o://c2457ecba97a133d5d6f55f42c7387d3e288e8680d310a89fa83725e8fac8ae1" gracePeriod=30 Oct 31 00:42:41.652401 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:41.652369 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 31 00:42:41.860836 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:41.860804 5939 generic.go:356] "Generic (PLEG): container finished" podID="5a453380-7b94-406a-a376-489b0709d0bd" containerID="c2457ecba97a133d5d6f55f42c7387d3e288e8680d310a89fa83725e8fac8ae1" exitCode=0 Oct 31 00:42:41.861190 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:41.860846 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" event={"ID":"5a453380-7b94-406a-a376-489b0709d0bd","Type":"ContainerDied","Data":"c2457ecba97a133d5d6f55f42c7387d3e288e8680d310a89fa83725e8fac8ae1"} Oct 31 00:42:42.648020 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:42.647979 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 31 00:42:42.648380 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:42.648356 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:42:43.450261 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:43.450220 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused" Oct 31 00:42:45.553576 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.553550 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:42:45.673889 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.673824 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a453380-7b94-406a-a376-489b0709d0bd-kube-api-access-tmd2d\" (UniqueName: \"kubernetes.io/projected/5a453380-7b94-406a-a376-489b0709d0bd-kube-api-access-tmd2d\") pod \"5a453380-7b94-406a-a376-489b0709d0bd\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " Oct 31 00:42:45.673889 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.673860 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a453380-7b94-406a-a376-489b0709d0bd-proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a453380-7b94-406a-a376-489b0709d0bd-proxy-tls\") pod \"5a453380-7b94-406a-a376-489b0709d0bd\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " Oct 31 00:42:45.674097 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.673901 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a453380-7b94-406a-a376-489b0709d0bd-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a453380-7b94-406a-a376-489b0709d0bd-cabundle-cert\") pod \"5a453380-7b94-406a-a376-489b0709d0bd\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " Oct 31 00:42:45.674097 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.673937 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a453380-7b94-406a-a376-489b0709d0bd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a453380-7b94-406a-a376-489b0709d0bd-kube-rbac-proxy-sar-config\") pod \"5a453380-7b94-406a-a376-489b0709d0bd\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " Oct 31 00:42:45.674097 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.673983 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a453380-7b94-406a-a376-489b0709d0bd-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a453380-7b94-406a-a376-489b0709d0bd-kserve-provision-location\") pod \"5a453380-7b94-406a-a376-489b0709d0bd\" (UID: \"5a453380-7b94-406a-a376-489b0709d0bd\") " Oct 31 00:42:45.674338 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.674311 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a453380-7b94-406a-a376-489b0709d0bd-kserve-provision-location" pod "5a453380-7b94-406a-a376-489b0709d0bd" (UID: "5a453380-7b94-406a-a376-489b0709d0bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:45.674338 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.674317 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a453380-7b94-406a-a376-489b0709d0bd-cabundle-cert" pod "5a453380-7b94-406a-a376-489b0709d0bd" (UID: "5a453380-7b94-406a-a376-489b0709d0bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:45.674505 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.674313 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a453380-7b94-406a-a376-489b0709d0bd-kube-rbac-proxy-sar-config" pod "5a453380-7b94-406a-a376-489b0709d0bd" (UID: "5a453380-7b94-406a-a376-489b0709d0bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:45.676042 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.676016 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a453380-7b94-406a-a376-489b0709d0bd-kube-api-access-tmd2d" pod "5a453380-7b94-406a-a376-489b0709d0bd" (UID: "5a453380-7b94-406a-a376-489b0709d0bd"). InnerVolumeSpecName "kube-api-access-tmd2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:45.676141 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.676072 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a453380-7b94-406a-a376-489b0709d0bd-proxy-tls" pod "5a453380-7b94-406a-a376-489b0709d0bd" (UID: "5a453380-7b94-406a-a376-489b0709d0bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:45.774973 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.774939 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a453380-7b94-406a-a376-489b0709d0bd-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:45.774973 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.774965 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmd2d\" (UniqueName: \"kubernetes.io/projected/5a453380-7b94-406a-a376-489b0709d0bd-kube-api-access-tmd2d\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:45.774973 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.774979 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a453380-7b94-406a-a376-489b0709d0bd-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:45.775202 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.774992 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a453380-7b94-406a-a376-489b0709d0bd-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:45.775202 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.775004 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a453380-7b94-406a-a376-489b0709d0bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:45.880918 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.880877 5939 generic.go:356] "Generic (PLEG): container finished" podID="5a453380-7b94-406a-a376-489b0709d0bd" containerID="73a4b5b28aee8cdca46acd83bc18a72c12279e6e3a22a4cc422cb438c1ed08ac" exitCode=0 Oct 31 00:42:45.881077 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.880942 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" Oct 31 00:42:45.881077 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.880955 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" event={"ID":"5a453380-7b94-406a-a376-489b0709d0bd","Type":"ContainerDied","Data":"73a4b5b28aee8cdca46acd83bc18a72c12279e6e3a22a4cc422cb438c1ed08ac"} Oct 31 00:42:45.881077 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.881015 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp" event={"ID":"5a453380-7b94-406a-a376-489b0709d0bd","Type":"ContainerDied","Data":"82d6e85f64b936e9377beaf8e0f638ad8ca37cde11b8c30b3fa681e89c3eccb3"} Oct 31 00:42:45.881077 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.881042 5939 scope.go:122] "RemoveContainer" containerID="c2457ecba97a133d5d6f55f42c7387d3e288e8680d310a89fa83725e8fac8ae1" Oct 31 00:42:45.889798 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.889780 5939 scope.go:122] "RemoveContainer" containerID="73a4b5b28aee8cdca46acd83bc18a72c12279e6e3a22a4cc422cb438c1ed08ac" Oct 31 00:42:45.897382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.897368 5939 scope.go:122] "RemoveContainer" containerID="7e272f057ee94296ec3f8e6246fe408ce4edf9f5f25aae7a89e9344d80d4d636" Oct 31 00:42:45.904826 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.904810 5939 scope.go:122] "RemoveContainer" containerID="c2457ecba97a133d5d6f55f42c7387d3e288e8680d310a89fa83725e8fac8ae1" Oct 31 00:42:45.905100 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:42:45.905076 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2457ecba97a133d5d6f55f42c7387d3e288e8680d310a89fa83725e8fac8ae1\": container with ID starting with c2457ecba97a133d5d6f55f42c7387d3e288e8680d310a89fa83725e8fac8ae1 not found: ID does not exist" containerID="c2457ecba97a133d5d6f55f42c7387d3e288e8680d310a89fa83725e8fac8ae1" Oct 31 00:42:45.905180 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.905109 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2457ecba97a133d5d6f55f42c7387d3e288e8680d310a89fa83725e8fac8ae1"} err="failed to get container status \"c2457ecba97a133d5d6f55f42c7387d3e288e8680d310a89fa83725e8fac8ae1\": rpc error: code = NotFound desc = could not find container \"c2457ecba97a133d5d6f55f42c7387d3e288e8680d310a89fa83725e8fac8ae1\": container with ID starting with c2457ecba97a133d5d6f55f42c7387d3e288e8680d310a89fa83725e8fac8ae1 not found: ID does not exist" Oct 31 00:42:45.905180 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.905124 5939 scope.go:122] "RemoveContainer" containerID="73a4b5b28aee8cdca46acd83bc18a72c12279e6e3a22a4cc422cb438c1ed08ac" Oct 31 00:42:45.905576 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:42:45.905554 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a4b5b28aee8cdca46acd83bc18a72c12279e6e3a22a4cc422cb438c1ed08ac\": container with ID starting with 73a4b5b28aee8cdca46acd83bc18a72c12279e6e3a22a4cc422cb438c1ed08ac not found: ID does not exist" containerID="73a4b5b28aee8cdca46acd83bc18a72c12279e6e3a22a4cc422cb438c1ed08ac" Oct 31 00:42:45.905696 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.905586 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a4b5b28aee8cdca46acd83bc18a72c12279e6e3a22a4cc422cb438c1ed08ac"} err="failed to get container status \"73a4b5b28aee8cdca46acd83bc18a72c12279e6e3a22a4cc422cb438c1ed08ac\": rpc error: code = NotFound desc = could not find container \"73a4b5b28aee8cdca46acd83bc18a72c12279e6e3a22a4cc422cb438c1ed08ac\": container with ID starting with 73a4b5b28aee8cdca46acd83bc18a72c12279e6e3a22a4cc422cb438c1ed08ac not found: ID does not exist" Oct 31 00:42:45.905696 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.905617 5939 scope.go:122] "RemoveContainer" containerID="7e272f057ee94296ec3f8e6246fe408ce4edf9f5f25aae7a89e9344d80d4d636" Oct 31 00:42:45.906435 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:42:45.906414 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e272f057ee94296ec3f8e6246fe408ce4edf9f5f25aae7a89e9344d80d4d636\": container with ID starting with 7e272f057ee94296ec3f8e6246fe408ce4edf9f5f25aae7a89e9344d80d4d636 not found: ID does not exist" containerID="7e272f057ee94296ec3f8e6246fe408ce4edf9f5f25aae7a89e9344d80d4d636" Oct 31 00:42:45.906505 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.906441 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e272f057ee94296ec3f8e6246fe408ce4edf9f5f25aae7a89e9344d80d4d636"} err="failed to get container status \"7e272f057ee94296ec3f8e6246fe408ce4edf9f5f25aae7a89e9344d80d4d636\": rpc error: code = NotFound desc = could not find container \"7e272f057ee94296ec3f8e6246fe408ce4edf9f5f25aae7a89e9344d80d4d636\": container with ID starting with 7e272f057ee94296ec3f8e6246fe408ce4edf9f5f25aae7a89e9344d80d4d636 not found: ID does not exist" Oct 31 00:42:45.907125 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.907105 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp"] Oct 31 00:42:45.915124 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:45.915104 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-pmml-v2-kserve-predictor-8547f8b69-fwqxp"] Oct 31 00:42:47.138801 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.138769 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6"] Oct 31 00:42:47.139158 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.139133 5939 cpu_manager.go:414] "containerMap: removing container" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="storage-initializer" Oct 31 00:42:47.139158 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.139146 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="storage-initializer" Oct 31 00:42:47.139158 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.139157 5939 cpu_manager.go:414] "containerMap: removing container" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kube-rbac-proxy" Oct 31 00:42:47.139299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.139164 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kube-rbac-proxy" Oct 31 00:42:47.139299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.139175 5939 cpu_manager.go:414] "containerMap: removing container" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kserve-container" Oct 31 00:42:47.139299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.139183 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kserve-container" Oct 31 00:42:47.139299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.139270 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kube-rbac-proxy" Oct 31 00:42:47.139299 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.139282 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a453380-7b94-406a-a376-489b0709d0bd" containerName="kserve-container" Oct 31 00:42:47.144110 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.144089 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:47.150945 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.150924 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 00:42:47.151511 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.151485 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-primary-9637c2-kube-rbac-proxy-sar-config\"" Oct 31 00:42:47.151618 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.151513 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:42:47.151618 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.151528 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 00:42:47.151618 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.151545 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 00:42:47.151618 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.151592 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-primary-9637c2-predictor-serving-cert\"" Oct 31 00:42:47.151787 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.151645 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 00:42:47.155348 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.155328 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6"] Oct 31 00:42:47.187638 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.187611 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cddw\" (UniqueName: \"kubernetes.io/projected/278566d1-57a7-40eb-8922-91c4379e420b-kube-api-access-8cddw\") pod \"isvc-primary-9637c2-predictor-5cc9d74658-z47v6\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:47.187754 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.187642 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/278566d1-57a7-40eb-8922-91c4379e420b-kserve-provision-location\") pod \"isvc-primary-9637c2-predictor-5cc9d74658-z47v6\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:47.187754 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.187680 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/278566d1-57a7-40eb-8922-91c4379e420b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9637c2-predictor-5cc9d74658-z47v6\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:47.187834 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.187811 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/278566d1-57a7-40eb-8922-91c4379e420b-cabundle-cert\") pod \"isvc-primary-9637c2-predictor-5cc9d74658-z47v6\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:47.187867 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.187834 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/278566d1-57a7-40eb-8922-91c4379e420b-proxy-tls\") pod \"isvc-primary-9637c2-predictor-5cc9d74658-z47v6\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:47.288321 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.288293 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cddw\" (UniqueName: \"kubernetes.io/projected/278566d1-57a7-40eb-8922-91c4379e420b-kube-api-access-8cddw\") pod \"isvc-primary-9637c2-predictor-5cc9d74658-z47v6\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:47.288482 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.288327 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/278566d1-57a7-40eb-8922-91c4379e420b-kserve-provision-location\") pod \"isvc-primary-9637c2-predictor-5cc9d74658-z47v6\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:47.288482 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.288351 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/278566d1-57a7-40eb-8922-91c4379e420b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9637c2-predictor-5cc9d74658-z47v6\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:47.288602 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.288550 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/278566d1-57a7-40eb-8922-91c4379e420b-cabundle-cert\") pod \"isvc-primary-9637c2-predictor-5cc9d74658-z47v6\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:47.288602 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.288591 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/278566d1-57a7-40eb-8922-91c4379e420b-proxy-tls\") pod \"isvc-primary-9637c2-predictor-5cc9d74658-z47v6\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:47.288828 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.288808 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/278566d1-57a7-40eb-8922-91c4379e420b-kserve-provision-location\") pod \"isvc-primary-9637c2-predictor-5cc9d74658-z47v6\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:47.289184 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.289151 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/278566d1-57a7-40eb-8922-91c4379e420b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9637c2-predictor-5cc9d74658-z47v6\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:47.289293 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.289188 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/278566d1-57a7-40eb-8922-91c4379e420b-cabundle-cert\") pod \"isvc-primary-9637c2-predictor-5cc9d74658-z47v6\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:47.291021 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.290999 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/278566d1-57a7-40eb-8922-91c4379e420b-proxy-tls\") pod \"isvc-primary-9637c2-predictor-5cc9d74658-z47v6\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:47.303094 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.303071 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cddw\" (UniqueName: \"kubernetes.io/projected/278566d1-57a7-40eb-8922-91c4379e420b-kube-api-access-8cddw\") pod \"isvc-primary-9637c2-predictor-5cc9d74658-z47v6\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:47.462725 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.462629 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:47.652051 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.652017 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a453380-7b94-406a-a376-489b0709d0bd" path="/var/lib/kubelet/pods/5a453380-7b94-406a-a376-489b0709d0bd/volumes" Oct 31 00:42:47.747210 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.747187 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6"] Oct 31 00:42:47.750730 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:42:47.750703 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278566d1_57a7_40eb_8922_91c4379e420b.slice/crio-7d54234e70ddc14ab3e84c2dc2ac76f93429ca96fc66b870b3eda051399ebb1a WatchSource:0}: Error finding container 7d54234e70ddc14ab3e84c2dc2ac76f93429ca96fc66b870b3eda051399ebb1a: Status 404 returned error can't find the container with id 7d54234e70ddc14ab3e84c2dc2ac76f93429ca96fc66b870b3eda051399ebb1a Oct 31 00:42:47.755493 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.755461 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 00:42:47.893286 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.893249 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" event={"ID":"278566d1-57a7-40eb-8922-91c4379e420b","Type":"ContainerStarted","Data":"116bfed2cf7a8f72b3747cf8b091e8f0d7990a50e565a5832e5e479bcbd2c2ff"} Oct 31 00:42:47.893434 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:47.893349 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" event={"ID":"278566d1-57a7-40eb-8922-91c4379e420b","Type":"ContainerStarted","Data":"7d54234e70ddc14ab3e84c2dc2ac76f93429ca96fc66b870b3eda051399ebb1a"} Oct 31 00:42:48.449999 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:48.449957 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused" Oct 31 00:42:48.450422 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:48.450090 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:42:51.911616 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:51.911583 5939 generic.go:356] "Generic (PLEG): container finished" podID="278566d1-57a7-40eb-8922-91c4379e420b" containerID="116bfed2cf7a8f72b3747cf8b091e8f0d7990a50e565a5832e5e479bcbd2c2ff" exitCode=0 Oct 31 00:42:51.912150 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:51.911680 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" event={"ID":"278566d1-57a7-40eb-8922-91c4379e420b","Type":"ContainerDied","Data":"116bfed2cf7a8f72b3747cf8b091e8f0d7990a50e565a5832e5e479bcbd2c2ff"} Oct 31 00:42:52.647549 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:52.647506 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 31 00:42:52.647893 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:52.647871 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:42:52.918961 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:52.918879 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" event={"ID":"278566d1-57a7-40eb-8922-91c4379e420b","Type":"ContainerStarted","Data":"4e18f27c56f3b3faaef002f614c3cc112ef4a74c97db67826039018ce6411389"} Oct 31 00:42:52.918961 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:52.918942 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" event={"ID":"278566d1-57a7-40eb-8922-91c4379e420b","Type":"ContainerStarted","Data":"fcdc492f5eaadecd1971a125ee21d0ca797d62669ef1016f1562b47cd307dc20"} Oct 31 00:42:52.919402 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:52.919221 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:52.919402 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:52.919374 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:52.920620 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:52.920593 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:42:52.948147 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:52.948090 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" podStartSLOduration=2.277256211 podStartE2EDuration="5.94807668s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.670820469s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:52.946412086 +0000 UTC m=+785.435687264" watchObservedRunningTime="2026-10-31 00:42:52.94807668 +0000 UTC m=+785.437351858" Oct 31 00:42:53.449478 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:53.449432 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused" Oct 31 00:42:53.922992 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:53.922948 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:42:58.449526 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:58.449481 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused" Oct 31 00:42:58.927065 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:58.927037 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:42:58.927526 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:42:58.927488 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:43:00.509359 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:43:00.509323 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:43:02.647781 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:02.647725 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 31 00:43:02.648276 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:02.647873 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:43:02.648276 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:02.648082 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:02.648276 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:02.648177 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:43:03.450255 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:03.450216 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused" Oct 31 00:43:05.975572 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:05.975490 5939 generic.go:356] "Generic (PLEG): container finished" podID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerID="512c9a739830ad3ebe76af44c29ef80b01c51617800dd5a6ad9db14e42c7e953" exitCode=0 Oct 31 00:43:05.975572 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:05.975560 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" event={"ID":"daa8cd3f-efe6-4524-a192-6888d26e6a05","Type":"ContainerDied","Data":"512c9a739830ad3ebe76af44c29ef80b01c51617800dd5a6ad9db14e42c7e953"} Oct 31 00:43:06.321134 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.321103 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:43:06.454176 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.454145 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa8cd3f-efe6-4524-a192-6888d26e6a05-kube-api-access-kwzbk\" (UniqueName: \"kubernetes.io/projected/daa8cd3f-efe6-4524-a192-6888d26e6a05-kube-api-access-kwzbk\") pod \"daa8cd3f-efe6-4524-a192-6888d26e6a05\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " Oct 31 00:43:06.454365 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.454190 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daa8cd3f-efe6-4524-a192-6888d26e6a05-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa8cd3f-efe6-4524-a192-6888d26e6a05-kserve-provision-location\") pod \"daa8cd3f-efe6-4524-a192-6888d26e6a05\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " Oct 31 00:43:06.454365 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.454223 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daa8cd3f-efe6-4524-a192-6888d26e6a05-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa8cd3f-efe6-4524-a192-6888d26e6a05-cabundle-cert\") pod \"daa8cd3f-efe6-4524-a192-6888d26e6a05\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " Oct 31 00:43:06.454365 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.454286 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daa8cd3f-efe6-4524-a192-6888d26e6a05-proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa8cd3f-efe6-4524-a192-6888d26e6a05-proxy-tls\") pod \"daa8cd3f-efe6-4524-a192-6888d26e6a05\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " Oct 31 00:43:06.454365 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.454309 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daa8cd3f-efe6-4524-a192-6888d26e6a05-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa8cd3f-efe6-4524-a192-6888d26e6a05-kube-rbac-proxy-sar-config\") pod \"daa8cd3f-efe6-4524-a192-6888d26e6a05\" (UID: \"daa8cd3f-efe6-4524-a192-6888d26e6a05\") " Oct 31 00:43:06.454614 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.454589 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa8cd3f-efe6-4524-a192-6888d26e6a05-kserve-provision-location" pod "daa8cd3f-efe6-4524-a192-6888d26e6a05" (UID: "daa8cd3f-efe6-4524-a192-6888d26e6a05"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:06.454685 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.454635 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa8cd3f-efe6-4524-a192-6888d26e6a05-cabundle-cert" pod "daa8cd3f-efe6-4524-a192-6888d26e6a05" (UID: "daa8cd3f-efe6-4524-a192-6888d26e6a05"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:06.454726 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.454700 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa8cd3f-efe6-4524-a192-6888d26e6a05-kube-rbac-proxy-sar-config" pod "daa8cd3f-efe6-4524-a192-6888d26e6a05" (UID: "daa8cd3f-efe6-4524-a192-6888d26e6a05"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:06.456472 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.456444 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa8cd3f-efe6-4524-a192-6888d26e6a05-kube-api-access-kwzbk" pod "daa8cd3f-efe6-4524-a192-6888d26e6a05" (UID: "daa8cd3f-efe6-4524-a192-6888d26e6a05"). InnerVolumeSpecName "kube-api-access-kwzbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:06.456472 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.456450 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa8cd3f-efe6-4524-a192-6888d26e6a05-proxy-tls" pod "daa8cd3f-efe6-4524-a192-6888d26e6a05" (UID: "daa8cd3f-efe6-4524-a192-6888d26e6a05"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:06.555790 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.555707 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwzbk\" (UniqueName: \"kubernetes.io/projected/daa8cd3f-efe6-4524-a192-6888d26e6a05-kube-api-access-kwzbk\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:06.555790 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.555735 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa8cd3f-efe6-4524-a192-6888d26e6a05-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:06.555790 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.555747 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa8cd3f-efe6-4524-a192-6888d26e6a05-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:06.555790 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.555758 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa8cd3f-efe6-4524-a192-6888d26e6a05-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:06.555790 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.555767 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa8cd3f-efe6-4524-a192-6888d26e6a05-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:06.988715 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.988690 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" Oct 31 00:43:06.989178 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.988686 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw" event={"ID":"daa8cd3f-efe6-4524-a192-6888d26e6a05","Type":"ContainerDied","Data":"b41af9749e8f86f46fdcddbdd46104dbcb3bba5d4eb2cd40489334fc7b2a2c9e"} Oct 31 00:43:06.989178 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.988852 5939 scope.go:122] "RemoveContainer" containerID="512c9a739830ad3ebe76af44c29ef80b01c51617800dd5a6ad9db14e42c7e953" Oct 31 00:43:06.997111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:06.997086 5939 scope.go:122] "RemoveContainer" containerID="46ea9c0af6a1d97bb4bc56ddf90cc7ae85effa6a4f25ef95b63e46df44e2765c" Oct 31 00:43:07.005061 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.005041 5939 scope.go:122] "RemoveContainer" containerID="d8fce6ac31aec5ff2f68ba48c667f71e205b7e00aa75a38e9978015d4307abe9" Oct 31 00:43:07.011712 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.011688 5939 scope.go:122] "RemoveContainer" containerID="b9b3bb4da830bc8de2ed61378fe9b672b6dce73462f1bda74398359c92e40e3b" Oct 31 00:43:07.015582 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.015559 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw"] Oct 31 00:43:07.023853 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.023829 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-predictor-7555846756-ntwzw"] Oct 31 00:43:07.432025 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.431996 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6"] Oct 31 00:43:07.432369 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.432354 5939 cpu_manager.go:414] "containerMap: removing container" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kserve-container" Oct 31 00:43:07.432419 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.432371 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kserve-container" Oct 31 00:43:07.432419 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.432385 5939 cpu_manager.go:414] "containerMap: removing container" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="storage-initializer" Oct 31 00:43:07.432419 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.432391 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="storage-initializer" Oct 31 00:43:07.432419 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.432401 5939 cpu_manager.go:414] "containerMap: removing container" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="agent" Oct 31 00:43:07.432419 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.432407 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="agent" Oct 31 00:43:07.432560 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.432442 5939 cpu_manager.go:414] "containerMap: removing container" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kube-rbac-proxy" Oct 31 00:43:07.432560 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.432448 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kube-rbac-proxy" Oct 31 00:43:07.432560 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.432500 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="agent" Oct 31 00:43:07.432560 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.432515 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kube-rbac-proxy" Oct 31 00:43:07.432560 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.432523 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" containerName="kserve-container" Oct 31 00:43:07.437556 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.437538 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:07.443911 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.443884 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\"" Oct 31 00:43:07.443911 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.443891 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:43:07.444117 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.443911 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\"" Oct 31 00:43:07.444117 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.444031 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\"" Oct 31 00:43:07.444117 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.444055 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\"" Oct 31 00:43:07.444331 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.444312 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\"" Oct 31 00:43:07.444466 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.444359 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\"" Oct 31 00:43:07.449459 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.449438 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6"] Oct 31 00:43:07.564291 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.564239 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4721086b-ea72-46e3-8569-cbc34d39fa0e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:07.564291 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.564296 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4721086b-ea72-46e3-8569-cbc34d39fa0e-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:07.564537 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.564377 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4721086b-ea72-46e3-8569-cbc34d39fa0e-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:07.564537 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.564429 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4721086b-ea72-46e3-8569-cbc34d39fa0e-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:07.564537 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.564453 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpcgv\" (UniqueName: \"kubernetes.io/projected/4721086b-ea72-46e3-8569-cbc34d39fa0e-kube-api-access-zpcgv\") pod \"isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:07.652278 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.652236 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daa8cd3f-efe6-4524-a192-6888d26e6a05" path="/var/lib/kubelet/pods/daa8cd3f-efe6-4524-a192-6888d26e6a05/volumes" Oct 31 00:43:07.665553 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.665529 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4721086b-ea72-46e3-8569-cbc34d39fa0e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:07.665655 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.665566 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4721086b-ea72-46e3-8569-cbc34d39fa0e-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:07.665655 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.665607 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4721086b-ea72-46e3-8569-cbc34d39fa0e-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:07.665655 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.665633 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4721086b-ea72-46e3-8569-cbc34d39fa0e-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:07.665831 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.665661 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpcgv\" (UniqueName: \"kubernetes.io/projected/4721086b-ea72-46e3-8569-cbc34d39fa0e-kube-api-access-zpcgv\") pod \"isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:07.666115 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.666087 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4721086b-ea72-46e3-8569-cbc34d39fa0e-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:07.666319 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.666300 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4721086b-ea72-46e3-8569-cbc34d39fa0e-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:07.666367 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.666308 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4721086b-ea72-46e3-8569-cbc34d39fa0e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:07.668147 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.668120 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4721086b-ea72-46e3-8569-cbc34d39fa0e-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:07.680452 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.680427 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpcgv\" (UniqueName: \"kubernetes.io/projected/4721086b-ea72-46e3-8569-cbc34d39fa0e-kube-api-access-zpcgv\") pod \"isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:07.758200 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:07.758107 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:08.045235 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:08.044843 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6"] Oct 31 00:43:08.047799 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:43:08.047763 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4721086b_ea72_46e3_8569_cbc34d39fa0e.slice/crio-34bab6fb53cd99f2570191f0997d7cec113bf39cae742bbcfeee0af2608c831b WatchSource:0}: Error finding container 34bab6fb53cd99f2570191f0997d7cec113bf39cae742bbcfeee0af2608c831b: Status 404 returned error can't find the container with id 34bab6fb53cd99f2570191f0997d7cec113bf39cae742bbcfeee0af2608c831b Oct 31 00:43:08.928083 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:08.928045 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:43:08.999304 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:08.999261 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" event={"ID":"4721086b-ea72-46e3-8569-cbc34d39fa0e","Type":"ContainerStarted","Data":"dbb32854b6020b6bbffe5699625893b19e899a26e4aa169225d143caba3ef3d0"} Oct 31 00:43:08.999498 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:08.999368 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" event={"ID":"4721086b-ea72-46e3-8569-cbc34d39fa0e","Type":"ContainerStarted","Data":"34bab6fb53cd99f2570191f0997d7cec113bf39cae742bbcfeee0af2608c831b"} Oct 31 00:43:13.017589 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:13.017549 5939 generic.go:356] "Generic (PLEG): container finished" podID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerID="dbb32854b6020b6bbffe5699625893b19e899a26e4aa169225d143caba3ef3d0" exitCode=0 Oct 31 00:43:13.018110 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:13.017623 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" event={"ID":"4721086b-ea72-46e3-8569-cbc34d39fa0e","Type":"ContainerDied","Data":"dbb32854b6020b6bbffe5699625893b19e899a26e4aa169225d143caba3ef3d0"} Oct 31 00:43:14.024434 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:14.024398 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" event={"ID":"4721086b-ea72-46e3-8569-cbc34d39fa0e","Type":"ContainerStarted","Data":"9025c86a8a84dd674af232b841e5f7e0eb73a5672b51f8bf540378f90942afdb"} Oct 31 00:43:14.024919 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:14.024470 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" event={"ID":"4721086b-ea72-46e3-8569-cbc34d39fa0e","Type":"ContainerStarted","Data":"9e1dd6bb1cd2977367a832bbe7a3eafba1e99bac4583e896ecd7f98fcdf01fbb"} Oct 31 00:43:14.024919 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:14.024497 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" event={"ID":"4721086b-ea72-46e3-8569-cbc34d39fa0e","Type":"ContainerStarted","Data":"9b456590b6ae367ac21cb7b4dfaf400ef369728185fbbcd57abf8d0aa41956a6"} Oct 31 00:43:14.024919 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:14.024779 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:14.025077 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:14.024943 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:14.026138 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:14.026110 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused" Oct 31 00:43:14.055388 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:14.055332 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podStartSLOduration=2.9019595259999997 podStartE2EDuration="7.055318648s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.153359122s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:14.054526616 +0000 UTC m=+806.543801795" watchObservedRunningTime="2026-10-31 00:43:14.055318648 +0000 UTC m=+806.544593894" Oct 31 00:43:15.028472 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:15.028439 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:15.028867 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:15.028581 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused" Oct 31 00:43:15.029717 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:15.029692 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:16.033035 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:16.032989 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused" Oct 31 00:43:16.033590 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:16.033568 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:18.928146 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:18.928107 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:43:21.038108 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:21.038078 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:43:21.038734 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:21.038703 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused" Oct 31 00:43:21.039167 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:21.039145 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:28.927758 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:28.927679 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:43:30.509473 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:43:30.509438 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:43:31.039493 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:31.039450 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused" Oct 31 00:43:31.039953 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:31.039927 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:38.927880 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:38.927839 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:43:41.038695 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:41.038641 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused" Oct 31 00:43:41.039124 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:41.039094 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:48.927937 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:48.927894 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:43:51.039040 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:51.039000 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused" Oct 31 00:43:51.039528 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:51.039410 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:58.928274 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:43:58.928230 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:44:00.509099 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:44:00.509054 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:44:01.039140 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:01.039098 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused" Oct 31 00:44:01.041344 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:01.039504 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:44:04.648059 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:04.648016 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:44:11.039503 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:11.039460 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused" Oct 31 00:44:11.040005 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:11.039983 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:44:14.648767 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:14.648735 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:44:21.039091 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:21.039051 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused" Oct 31 00:44:21.039638 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:21.039577 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:44:30.509235 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:44:30.509194 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:44:31.039416 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:31.039388 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:44:31.039606 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:31.039518 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:44:32.567488 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.567452 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl"] Oct 31 00:44:32.571398 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.571381 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:32.581159 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.581136 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"invalid-s3-sa-9637c2-dockercfg-5dflv\"" Oct 31 00:44:32.581159 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.581136 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-secondary-9637c2-predictor-serving-cert\"" Oct 31 00:44:32.581305 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.581157 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-secondary-9637c2-kube-rbac-proxy-sar-config\"" Oct 31 00:44:32.581305 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.581179 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"invalid-s3-secret-9637c2\"" Oct 31 00:44:32.585153 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.585131 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl"] Oct 31 00:44:32.699997 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.699957 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx57s\" (UniqueName: \"kubernetes.io/projected/a17f9b78-762c-41b8-809e-dde9f384b847-kube-api-access-jx57s\") pod \"isvc-secondary-9637c2-predictor-5c64df9485-98dnl\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:32.700177 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.700018 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17f9b78-762c-41b8-809e-dde9f384b847-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9637c2-predictor-5c64df9485-98dnl\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:32.700177 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.700100 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17f9b78-762c-41b8-809e-dde9f384b847-proxy-tls\") pod \"isvc-secondary-9637c2-predictor-5c64df9485-98dnl\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:32.700177 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.700139 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17f9b78-762c-41b8-809e-dde9f384b847-kserve-provision-location\") pod \"isvc-secondary-9637c2-predictor-5c64df9485-98dnl\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:32.700371 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.700257 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a17f9b78-762c-41b8-809e-dde9f384b847-cabundle-cert\") pod \"isvc-secondary-9637c2-predictor-5c64df9485-98dnl\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:32.801456 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.801422 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17f9b78-762c-41b8-809e-dde9f384b847-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9637c2-predictor-5c64df9485-98dnl\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:32.801643 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.801461 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17f9b78-762c-41b8-809e-dde9f384b847-proxy-tls\") pod \"isvc-secondary-9637c2-predictor-5c64df9485-98dnl\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:32.801643 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.801491 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17f9b78-762c-41b8-809e-dde9f384b847-kserve-provision-location\") pod \"isvc-secondary-9637c2-predictor-5c64df9485-98dnl\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:32.801643 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.801554 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a17f9b78-762c-41b8-809e-dde9f384b847-cabundle-cert\") pod \"isvc-secondary-9637c2-predictor-5c64df9485-98dnl\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:32.801643 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.801619 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx57s\" (UniqueName: \"kubernetes.io/projected/a17f9b78-762c-41b8-809e-dde9f384b847-kube-api-access-jx57s\") pod \"isvc-secondary-9637c2-predictor-5c64df9485-98dnl\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:32.802042 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.802020 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17f9b78-762c-41b8-809e-dde9f384b847-kserve-provision-location\") pod \"isvc-secondary-9637c2-predictor-5c64df9485-98dnl\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:32.802246 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.802224 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17f9b78-762c-41b8-809e-dde9f384b847-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9637c2-predictor-5c64df9485-98dnl\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:32.802367 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.802350 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a17f9b78-762c-41b8-809e-dde9f384b847-cabundle-cert\") pod \"isvc-secondary-9637c2-predictor-5c64df9485-98dnl\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:32.804026 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.804008 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17f9b78-762c-41b8-809e-dde9f384b847-proxy-tls\") pod \"isvc-secondary-9637c2-predictor-5c64df9485-98dnl\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:32.816919 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.816898 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx57s\" (UniqueName: \"kubernetes.io/projected/a17f9b78-762c-41b8-809e-dde9f384b847-kube-api-access-jx57s\") pod \"isvc-secondary-9637c2-predictor-5c64df9485-98dnl\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:32.889681 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:32.889642 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:33.170968 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:33.170940 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl"] Oct 31 00:44:33.173548 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:44:33.173502 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17f9b78_762c_41b8_809e_dde9f384b847.slice/crio-98ec459d52fb0b12cddd8ec8ccb897ed36408612ba0d4753a8f020328159ae46 WatchSource:0}: Error finding container 98ec459d52fb0b12cddd8ec8ccb897ed36408612ba0d4753a8f020328159ae46: Status 404 returned error can't find the container with id 98ec459d52fb0b12cddd8ec8ccb897ed36408612ba0d4753a8f020328159ae46 Oct 31 00:44:33.378779 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:33.378743 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" event={"ID":"a17f9b78-762c-41b8-809e-dde9f384b847","Type":"ContainerStarted","Data":"b54de94704db758ebc0110fdd0bb520c6a9fc09ff0967512cb0d14dbfce52781"} Oct 31 00:44:33.378942 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:33.378801 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" event={"ID":"a17f9b78-762c-41b8-809e-dde9f384b847","Type":"ContainerStarted","Data":"98ec459d52fb0b12cddd8ec8ccb897ed36408612ba0d4753a8f020328159ae46"} Oct 31 00:44:38.400771 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:38.400743 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-secondary-9637c2-predictor-5c64df9485-98dnl_a17f9b78-762c-41b8-809e-dde9f384b847/storage-initializer/0.log" Oct 31 00:44:38.401197 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:38.400783 5939 generic.go:356] "Generic (PLEG): container finished" podID="a17f9b78-762c-41b8-809e-dde9f384b847" containerID="b54de94704db758ebc0110fdd0bb520c6a9fc09ff0967512cb0d14dbfce52781" exitCode=1 Oct 31 00:44:38.401197 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:38.400861 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" event={"ID":"a17f9b78-762c-41b8-809e-dde9f384b847","Type":"ContainerDied","Data":"b54de94704db758ebc0110fdd0bb520c6a9fc09ff0967512cb0d14dbfce52781"} Oct 31 00:44:39.407370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:39.407343 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-secondary-9637c2-predictor-5c64df9485-98dnl_a17f9b78-762c-41b8-809e-dde9f384b847/storage-initializer/0.log" Oct 31 00:44:39.407772 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:39.407453 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" event={"ID":"a17f9b78-762c-41b8-809e-dde9f384b847","Type":"ContainerStarted","Data":"9f3b6f06ea68aa795784ac703a73566afbce88734248a2fac95a26cb5212e20e"} Oct 31 00:44:43.217489 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:43.217457 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6"] Oct 31 00:44:43.217945 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:43.217775 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kserve-container" containerID="cri-o://9b456590b6ae367ac21cb7b4dfaf400ef369728185fbbcd57abf8d0aa41956a6" gracePeriod=30 Oct 31 00:44:43.217945 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:43.217789 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="agent" containerID="cri-o://9025c86a8a84dd674af232b841e5f7e0eb73a5672b51f8bf540378f90942afdb" gracePeriod=30 Oct 31 00:44:43.217945 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:43.217819 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kube-rbac-proxy" containerID="cri-o://9e1dd6bb1cd2977367a832bbe7a3eafba1e99bac4583e896ecd7f98fcdf01fbb" gracePeriod=30 Oct 31 00:44:43.427186 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:43.427155 5939 generic.go:356] "Generic (PLEG): container finished" podID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerID="9e1dd6bb1cd2977367a832bbe7a3eafba1e99bac4583e896ecd7f98fcdf01fbb" exitCode=0 Oct 31 00:44:43.427371 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:43.427222 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" event={"ID":"4721086b-ea72-46e3-8569-cbc34d39fa0e","Type":"ContainerDied","Data":"9e1dd6bb1cd2977367a832bbe7a3eafba1e99bac4583e896ecd7f98fcdf01fbb"} Oct 31 00:44:45.436733 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:45.436707 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-secondary-9637c2-predictor-5c64df9485-98dnl_a17f9b78-762c-41b8-809e-dde9f384b847/storage-initializer/1.log" Oct 31 00:44:45.437145 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:45.437114 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-secondary-9637c2-predictor-5c64df9485-98dnl_a17f9b78-762c-41b8-809e-dde9f384b847/storage-initializer/0.log" Oct 31 00:44:45.437189 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:45.437142 5939 generic.go:356] "Generic (PLEG): container finished" podID="a17f9b78-762c-41b8-809e-dde9f384b847" containerID="9f3b6f06ea68aa795784ac703a73566afbce88734248a2fac95a26cb5212e20e" exitCode=1 Oct 31 00:44:45.437265 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:45.437242 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" event={"ID":"a17f9b78-762c-41b8-809e-dde9f384b847","Type":"ContainerDied","Data":"9f3b6f06ea68aa795784ac703a73566afbce88734248a2fac95a26cb5212e20e"} Oct 31 00:44:45.437324 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:45.437312 5939 scope.go:122] "RemoveContainer" containerID="b54de94704db758ebc0110fdd0bb520c6a9fc09ff0967512cb0d14dbfce52781" Oct 31 00:44:45.437676 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:45.437653 5939 scope.go:122] "RemoveContainer" containerID="b54de94704db758ebc0110fdd0bb520c6a9fc09ff0967512cb0d14dbfce52781" Oct 31 00:44:45.448485 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:44:45.448460 5939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-9637c2-predictor-5c64df9485-98dnl_e2e-gw1-1c69aea164(a17f9b78-762c-41b8-809e-dde9f384b847)\"" pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" podUID="a17f9b78-762c-41b8-809e-dde9f384b847" Oct 31 00:44:45.448715 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:44:45.448688 5939 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9637c2-predictor-5c64df9485-98dnl_e2e-gw1-1c69aea164_a17f9b78-762c-41b8-809e-dde9f384b847_0 in pod sandbox 98ec459d52fb0b12cddd8ec8ccb897ed36408612ba0d4753a8f020328159ae46 from index: no such id: 'b54de94704db758ebc0110fdd0bb520c6a9fc09ff0967512cb0d14dbfce52781'" containerID="b54de94704db758ebc0110fdd0bb520c6a9fc09ff0967512cb0d14dbfce52781" Oct 31 00:44:45.448791 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:45.448725 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b54de94704db758ebc0110fdd0bb520c6a9fc09ff0967512cb0d14dbfce52781"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9637c2-predictor-5c64df9485-98dnl_e2e-gw1-1c69aea164_a17f9b78-762c-41b8-809e-dde9f384b847_0 in pod sandbox 98ec459d52fb0b12cddd8ec8ccb897ed36408612ba0d4753a8f020328159ae46 from index: no such id: 'b54de94704db758ebc0110fdd0bb520c6a9fc09ff0967512cb0d14dbfce52781'" Oct 31 00:44:46.034050 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:46.034011 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" Oct 31 00:44:46.442926 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:46.442898 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-secondary-9637c2-predictor-5c64df9485-98dnl_a17f9b78-762c-41b8-809e-dde9f384b847/storage-initializer/1.log" Oct 31 00:44:47.726265 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:47.726240 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-secondary-9637c2-predictor-5c64df9485-98dnl_a17f9b78-762c-41b8-809e-dde9f384b847/storage-initializer/1.log" Oct 31 00:44:47.726655 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:47.726300 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-secondary-9637c2-predictor-5c64df9485-98dnl_a17f9b78-762c-41b8-809e-dde9f384b847/storage-initializer/1.log" Oct 31 00:44:47.754683 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:47.754651 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:44:47.755048 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:47.755027 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:44:48.453405 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:48.453372 5939 generic.go:356] "Generic (PLEG): container finished" podID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerID="9b456590b6ae367ac21cb7b4dfaf400ef369728185fbbcd57abf8d0aa41956a6" exitCode=0 Oct 31 00:44:48.453568 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:48.453449 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" event={"ID":"4721086b-ea72-46e3-8569-cbc34d39fa0e","Type":"ContainerDied","Data":"9b456590b6ae367ac21cb7b4dfaf400ef369728185fbbcd57abf8d0aa41956a6"} Oct 31 00:44:50.667924 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:50.667892 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl"] Oct 31 00:44:50.706039 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:50.706011 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6"] Oct 31 00:44:50.706274 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:50.706244 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kserve-container" containerID="cri-o://fcdc492f5eaadecd1971a125ee21d0ca797d62669ef1016f1562b47cd307dc20" gracePeriod=30 Oct 31 00:44:50.706374 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:50.706264 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kube-rbac-proxy" containerID="cri-o://4e18f27c56f3b3faaef002f614c3cc112ef4a74c97db67826039018ce6411389" gracePeriod=30 Oct 31 00:44:50.827479 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:50.827453 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-secondary-9637c2-predictor-5c64df9485-98dnl_a17f9b78-762c-41b8-809e-dde9f384b847/storage-initializer/1.log" Oct 31 00:44:50.829567 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:50.829550 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:50.958164 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:50.958075 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a17f9b78-762c-41b8-809e-dde9f384b847-kube-api-access-jx57s\" (UniqueName: \"kubernetes.io/projected/a17f9b78-762c-41b8-809e-dde9f384b847-kube-api-access-jx57s\") pod \"a17f9b78-762c-41b8-809e-dde9f384b847\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " Oct 31 00:44:50.958164 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:50.958122 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a17f9b78-762c-41b8-809e-dde9f384b847-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17f9b78-762c-41b8-809e-dde9f384b847-kube-rbac-proxy-sar-config\") pod \"a17f9b78-762c-41b8-809e-dde9f384b847\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " Oct 31 00:44:50.958164 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:50.958145 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a17f9b78-762c-41b8-809e-dde9f384b847-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17f9b78-762c-41b8-809e-dde9f384b847-proxy-tls\") pod \"a17f9b78-762c-41b8-809e-dde9f384b847\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " Oct 31 00:44:50.958379 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:50.958218 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a17f9b78-762c-41b8-809e-dde9f384b847-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a17f9b78-762c-41b8-809e-dde9f384b847-cabundle-cert\") pod \"a17f9b78-762c-41b8-809e-dde9f384b847\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " Oct 31 00:44:50.958379 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:50.958265 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a17f9b78-762c-41b8-809e-dde9f384b847-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17f9b78-762c-41b8-809e-dde9f384b847-kserve-provision-location\") pod \"a17f9b78-762c-41b8-809e-dde9f384b847\" (UID: \"a17f9b78-762c-41b8-809e-dde9f384b847\") " Oct 31 00:44:50.958648 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:50.958623 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17f9b78-762c-41b8-809e-dde9f384b847-cabundle-cert" pod "a17f9b78-762c-41b8-809e-dde9f384b847" (UID: "a17f9b78-762c-41b8-809e-dde9f384b847"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:50.958648 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:50.958633 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17f9b78-762c-41b8-809e-dde9f384b847-kserve-provision-location" pod "a17f9b78-762c-41b8-809e-dde9f384b847" (UID: "a17f9b78-762c-41b8-809e-dde9f384b847"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:50.958800 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:50.958732 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17f9b78-762c-41b8-809e-dde9f384b847-kube-rbac-proxy-sar-config" pod "a17f9b78-762c-41b8-809e-dde9f384b847" (UID: "a17f9b78-762c-41b8-809e-dde9f384b847"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:50.960204 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:50.960181 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17f9b78-762c-41b8-809e-dde9f384b847-kube-api-access-jx57s" pod "a17f9b78-762c-41b8-809e-dde9f384b847" (UID: "a17f9b78-762c-41b8-809e-dde9f384b847"). InnerVolumeSpecName "kube-api-access-jx57s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:50.960204 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:50.960184 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a17f9b78-762c-41b8-809e-dde9f384b847-proxy-tls" pod "a17f9b78-762c-41b8-809e-dde9f384b847" (UID: "a17f9b78-762c-41b8-809e-dde9f384b847"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:51.033405 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:51.033361 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" Oct 31 00:44:51.038928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:51.038892 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused" Oct 31 00:44:51.039255 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:51.039233 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:44:51.059222 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:51.059202 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17f9b78-762c-41b8-809e-dde9f384b847-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:51.059222 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:51.059221 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17f9b78-762c-41b8-809e-dde9f384b847-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:51.059332 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:51.059231 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a17f9b78-762c-41b8-809e-dde9f384b847-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:51.059332 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:51.059239 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17f9b78-762c-41b8-809e-dde9f384b847-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:51.059332 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:51.059248 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx57s\" (UniqueName: \"kubernetes.io/projected/a17f9b78-762c-41b8-809e-dde9f384b847-kube-api-access-jx57s\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:51.467847 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:51.467821 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-secondary-9637c2-predictor-5c64df9485-98dnl_a17f9b78-762c-41b8-809e-dde9f384b847/storage-initializer/1.log" Oct 31 00:44:51.468053 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:51.467931 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" Oct 31 00:44:51.468053 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:51.467935 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl" event={"ID":"a17f9b78-762c-41b8-809e-dde9f384b847","Type":"ContainerDied","Data":"98ec459d52fb0b12cddd8ec8ccb897ed36408612ba0d4753a8f020328159ae46"} Oct 31 00:44:51.468053 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:51.468002 5939 scope.go:122] "RemoveContainer" containerID="9f3b6f06ea68aa795784ac703a73566afbce88734248a2fac95a26cb5212e20e" Oct 31 00:44:51.470244 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:51.470217 5939 generic.go:356] "Generic (PLEG): container finished" podID="278566d1-57a7-40eb-8922-91c4379e420b" containerID="4e18f27c56f3b3faaef002f614c3cc112ef4a74c97db67826039018ce6411389" exitCode=0 Oct 31 00:44:51.470349 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:51.470254 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" event={"ID":"278566d1-57a7-40eb-8922-91c4379e420b","Type":"ContainerDied","Data":"4e18f27c56f3b3faaef002f614c3cc112ef4a74c97db67826039018ce6411389"} Oct 31 00:44:51.514708 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:51.514681 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl"] Oct 31 00:44:51.521425 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:51.521393 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-secondary-9637c2-predictor-5c64df9485-98dnl"] Oct 31 00:44:51.652293 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:51.652249 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a17f9b78-762c-41b8-809e-dde9f384b847" path="/var/lib/kubelet/pods/a17f9b78-762c-41b8-809e-dde9f384b847/volumes" Oct 31 00:44:53.923171 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:53.923132 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused" Oct 31 00:44:54.648923 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:54.648881 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Oct 31 00:44:55.448906 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.448882 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:44:55.493809 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.493734 5939 generic.go:356] "Generic (PLEG): container finished" podID="278566d1-57a7-40eb-8922-91c4379e420b" containerID="fcdc492f5eaadecd1971a125ee21d0ca797d62669ef1016f1562b47cd307dc20" exitCode=0 Oct 31 00:44:55.493809 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.493795 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" Oct 31 00:44:55.493809 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.493791 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" event={"ID":"278566d1-57a7-40eb-8922-91c4379e420b","Type":"ContainerDied","Data":"fcdc492f5eaadecd1971a125ee21d0ca797d62669ef1016f1562b47cd307dc20"} Oct 31 00:44:55.493988 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.493864 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6" event={"ID":"278566d1-57a7-40eb-8922-91c4379e420b","Type":"ContainerDied","Data":"7d54234e70ddc14ab3e84c2dc2ac76f93429ca96fc66b870b3eda051399ebb1a"} Oct 31 00:44:55.493988 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.493902 5939 scope.go:122] "RemoveContainer" containerID="4e18f27c56f3b3faaef002f614c3cc112ef4a74c97db67826039018ce6411389" Oct 31 00:44:55.501871 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.501797 5939 scope.go:122] "RemoveContainer" containerID="fcdc492f5eaadecd1971a125ee21d0ca797d62669ef1016f1562b47cd307dc20" Oct 31 00:44:55.509043 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.509027 5939 scope.go:122] "RemoveContainer" containerID="116bfed2cf7a8f72b3747cf8b091e8f0d7990a50e565a5832e5e479bcbd2c2ff" Oct 31 00:44:55.516098 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.516078 5939 scope.go:122] "RemoveContainer" containerID="4e18f27c56f3b3faaef002f614c3cc112ef4a74c97db67826039018ce6411389" Oct 31 00:44:55.516392 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:44:55.516372 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e18f27c56f3b3faaef002f614c3cc112ef4a74c97db67826039018ce6411389\": container with ID starting with 4e18f27c56f3b3faaef002f614c3cc112ef4a74c97db67826039018ce6411389 not found: ID does not exist" containerID="4e18f27c56f3b3faaef002f614c3cc112ef4a74c97db67826039018ce6411389" Oct 31 00:44:55.516460 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.516400 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e18f27c56f3b3faaef002f614c3cc112ef4a74c97db67826039018ce6411389"} err="failed to get container status \"4e18f27c56f3b3faaef002f614c3cc112ef4a74c97db67826039018ce6411389\": rpc error: code = NotFound desc = could not find container \"4e18f27c56f3b3faaef002f614c3cc112ef4a74c97db67826039018ce6411389\": container with ID starting with 4e18f27c56f3b3faaef002f614c3cc112ef4a74c97db67826039018ce6411389 not found: ID does not exist" Oct 31 00:44:55.516460 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.516417 5939 scope.go:122] "RemoveContainer" containerID="fcdc492f5eaadecd1971a125ee21d0ca797d62669ef1016f1562b47cd307dc20" Oct 31 00:44:55.516702 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:44:55.516641 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcdc492f5eaadecd1971a125ee21d0ca797d62669ef1016f1562b47cd307dc20\": container with ID starting with fcdc492f5eaadecd1971a125ee21d0ca797d62669ef1016f1562b47cd307dc20 not found: ID does not exist" containerID="fcdc492f5eaadecd1971a125ee21d0ca797d62669ef1016f1562b47cd307dc20" Oct 31 00:44:55.516702 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.516693 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcdc492f5eaadecd1971a125ee21d0ca797d62669ef1016f1562b47cd307dc20"} err="failed to get container status \"fcdc492f5eaadecd1971a125ee21d0ca797d62669ef1016f1562b47cd307dc20\": rpc error: code = NotFound desc = could not find container \"fcdc492f5eaadecd1971a125ee21d0ca797d62669ef1016f1562b47cd307dc20\": container with ID starting with fcdc492f5eaadecd1971a125ee21d0ca797d62669ef1016f1562b47cd307dc20 not found: ID does not exist" Oct 31 00:44:55.516818 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.516708 5939 scope.go:122] "RemoveContainer" containerID="116bfed2cf7a8f72b3747cf8b091e8f0d7990a50e565a5832e5e479bcbd2c2ff" Oct 31 00:44:55.516956 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:44:55.516941 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"116bfed2cf7a8f72b3747cf8b091e8f0d7990a50e565a5832e5e479bcbd2c2ff\": container with ID starting with 116bfed2cf7a8f72b3747cf8b091e8f0d7990a50e565a5832e5e479bcbd2c2ff not found: ID does not exist" containerID="116bfed2cf7a8f72b3747cf8b091e8f0d7990a50e565a5832e5e479bcbd2c2ff" Oct 31 00:44:55.517000 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.516961 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"116bfed2cf7a8f72b3747cf8b091e8f0d7990a50e565a5832e5e479bcbd2c2ff"} err="failed to get container status \"116bfed2cf7a8f72b3747cf8b091e8f0d7990a50e565a5832e5e479bcbd2c2ff\": rpc error: code = NotFound desc = could not find container \"116bfed2cf7a8f72b3747cf8b091e8f0d7990a50e565a5832e5e479bcbd2c2ff\": container with ID starting with 116bfed2cf7a8f72b3747cf8b091e8f0d7990a50e565a5832e5e479bcbd2c2ff not found: ID does not exist" Oct 31 00:44:55.600557 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.600531 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278566d1-57a7-40eb-8922-91c4379e420b-kube-api-access-8cddw\" (UniqueName: \"kubernetes.io/projected/278566d1-57a7-40eb-8922-91c4379e420b-kube-api-access-8cddw\") pod \"278566d1-57a7-40eb-8922-91c4379e420b\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " Oct 31 00:44:55.600733 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.600562 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278566d1-57a7-40eb-8922-91c4379e420b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/278566d1-57a7-40eb-8922-91c4379e420b-kserve-provision-location\") pod \"278566d1-57a7-40eb-8922-91c4379e420b\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " Oct 31 00:44:55.600733 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.600591 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/278566d1-57a7-40eb-8922-91c4379e420b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/278566d1-57a7-40eb-8922-91c4379e420b-kube-rbac-proxy-sar-config\") pod \"278566d1-57a7-40eb-8922-91c4379e420b\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " Oct 31 00:44:55.600733 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.600617 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/278566d1-57a7-40eb-8922-91c4379e420b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/278566d1-57a7-40eb-8922-91c4379e420b-proxy-tls\") pod \"278566d1-57a7-40eb-8922-91c4379e420b\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " Oct 31 00:44:55.600733 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.600641 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/278566d1-57a7-40eb-8922-91c4379e420b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/278566d1-57a7-40eb-8922-91c4379e420b-cabundle-cert\") pod \"278566d1-57a7-40eb-8922-91c4379e420b\" (UID: \"278566d1-57a7-40eb-8922-91c4379e420b\") " Oct 31 00:44:55.601025 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.600967 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278566d1-57a7-40eb-8922-91c4379e420b-kserve-provision-location" pod "278566d1-57a7-40eb-8922-91c4379e420b" (UID: "278566d1-57a7-40eb-8922-91c4379e420b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:55.601025 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.600995 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/278566d1-57a7-40eb-8922-91c4379e420b-kube-rbac-proxy-sar-config" pod "278566d1-57a7-40eb-8922-91c4379e420b" (UID: "278566d1-57a7-40eb-8922-91c4379e420b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:55.601098 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.601041 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/278566d1-57a7-40eb-8922-91c4379e420b-cabundle-cert" pod "278566d1-57a7-40eb-8922-91c4379e420b" (UID: "278566d1-57a7-40eb-8922-91c4379e420b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:55.602596 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.602576 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/278566d1-57a7-40eb-8922-91c4379e420b-proxy-tls" pod "278566d1-57a7-40eb-8922-91c4379e420b" (UID: "278566d1-57a7-40eb-8922-91c4379e420b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:55.602677 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.602653 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278566d1-57a7-40eb-8922-91c4379e420b-kube-api-access-8cddw" pod "278566d1-57a7-40eb-8922-91c4379e420b" (UID: "278566d1-57a7-40eb-8922-91c4379e420b"). InnerVolumeSpecName "kube-api-access-8cddw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:55.702169 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.702144 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cddw\" (UniqueName: \"kubernetes.io/projected/278566d1-57a7-40eb-8922-91c4379e420b-kube-api-access-8cddw\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:55.702169 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.702166 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/278566d1-57a7-40eb-8922-91c4379e420b-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:55.702300 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.702176 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/278566d1-57a7-40eb-8922-91c4379e420b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:55.702300 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.702184 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/278566d1-57a7-40eb-8922-91c4379e420b-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:55.702300 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.702192 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/278566d1-57a7-40eb-8922-91c4379e420b-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:55.815963 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.815893 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6"] Oct 31 00:44:55.821066 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:55.821041 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-primary-9637c2-predictor-5cc9d74658-z47v6"] Oct 31 00:44:56.033255 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.033221 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" Oct 31 00:44:56.033419 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.033340 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:44:56.407381 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.407345 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv"] Oct 31 00:44:56.407702 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.407690 5939 cpu_manager.go:414] "containerMap: removing container" podUID="a17f9b78-762c-41b8-809e-dde9f384b847" containerName="storage-initializer" Oct 31 00:44:56.407756 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.407705 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a17f9b78-762c-41b8-809e-dde9f384b847" containerName="storage-initializer" Oct 31 00:44:56.407756 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.407717 5939 cpu_manager.go:414] "containerMap: removing container" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kube-rbac-proxy" Oct 31 00:44:56.407756 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.407722 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kube-rbac-proxy" Oct 31 00:44:56.407756 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.407736 5939 cpu_manager.go:414] "containerMap: removing container" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kserve-container" Oct 31 00:44:56.407756 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.407741 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kserve-container" Oct 31 00:44:56.407756 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.407748 5939 cpu_manager.go:414] "containerMap: removing container" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="storage-initializer" Oct 31 00:44:56.407756 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.407753 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="storage-initializer" Oct 31 00:44:56.407946 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.407823 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kserve-container" Oct 31 00:44:56.407946 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.407831 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="a17f9b78-762c-41b8-809e-dde9f384b847" containerName="storage-initializer" Oct 31 00:44:56.407946 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.407843 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="278566d1-57a7-40eb-8922-91c4379e420b" containerName="kube-rbac-proxy" Oct 31 00:44:56.407946 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.407898 5939 cpu_manager.go:414] "containerMap: removing container" podUID="a17f9b78-762c-41b8-809e-dde9f384b847" containerName="storage-initializer" Oct 31 00:44:56.407946 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.407902 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a17f9b78-762c-41b8-809e-dde9f384b847" containerName="storage-initializer" Oct 31 00:44:56.408104 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.407970 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="a17f9b78-762c-41b8-809e-dde9f384b847" containerName="storage-initializer" Oct 31 00:44:56.412478 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.412463 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:44:56.418591 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.418568 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"fail-s3-secret-5e43f7\"" Oct 31 00:44:56.418591 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.418586 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 00:44:56.418591 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.418577 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-init-fail-5e43f7-kube-rbac-proxy-sar-config\"" Oct 31 00:44:56.418850 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.418574 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:44:56.418850 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.418750 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"fail-s3-sa-5e43f7-dockercfg-j272k\"" Oct 31 00:44:56.418850 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.418757 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 00:44:56.418850 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.418794 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-init-fail-5e43f7-predictor-serving-cert\"" Oct 31 00:44:56.424146 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.424121 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv"] Oct 31 00:44:56.508936 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.508905 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b76867ad-cef7-4d05-a375-a2e01134301d-kserve-provision-location\") pod \"isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:44:56.509317 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.508958 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b76867ad-cef7-4d05-a375-a2e01134301d-cabundle-cert\") pod \"isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:44:56.509317 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.509034 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfrtv\" (UniqueName: \"kubernetes.io/projected/b76867ad-cef7-4d05-a375-a2e01134301d-kube-api-access-xfrtv\") pod \"isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:44:56.509317 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.509074 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b76867ad-cef7-4d05-a375-a2e01134301d-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:44:56.509317 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.509158 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b76867ad-cef7-4d05-a375-a2e01134301d-proxy-tls\") pod \"isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:44:56.610043 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.610012 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfrtv\" (UniqueName: \"kubernetes.io/projected/b76867ad-cef7-4d05-a375-a2e01134301d-kube-api-access-xfrtv\") pod \"isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:44:56.610192 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.610060 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b76867ad-cef7-4d05-a375-a2e01134301d-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:44:56.610192 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.610098 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b76867ad-cef7-4d05-a375-a2e01134301d-proxy-tls\") pod \"isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:44:56.610192 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.610137 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b76867ad-cef7-4d05-a375-a2e01134301d-kserve-provision-location\") pod \"isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:44:56.610368 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.610205 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b76867ad-cef7-4d05-a375-a2e01134301d-cabundle-cert\") pod \"isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:44:56.610617 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.610591 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b76867ad-cef7-4d05-a375-a2e01134301d-kserve-provision-location\") pod \"isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:44:56.610861 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.610842 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b76867ad-cef7-4d05-a375-a2e01134301d-cabundle-cert\") pod \"isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:44:56.610918 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.610843 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b76867ad-cef7-4d05-a375-a2e01134301d-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:44:56.612548 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.612528 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b76867ad-cef7-4d05-a375-a2e01134301d-proxy-tls\") pod \"isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:44:56.626127 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.626107 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfrtv\" (UniqueName: \"kubernetes.io/projected/b76867ad-cef7-4d05-a375-a2e01134301d-kube-api-access-xfrtv\") pod \"isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:44:56.730094 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:56.730036 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:44:57.010050 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:57.009966 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv"] Oct 31 00:44:57.012769 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:44:57.012737 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76867ad_cef7_4d05_a375_a2e01134301d.slice/crio-f1d2769bb5b03f7ca2599248079bd023f4948c9b26a06125f582625d71d5af5c WatchSource:0}: Error finding container f1d2769bb5b03f7ca2599248079bd023f4948c9b26a06125f582625d71d5af5c: Status 404 returned error can't find the container with id f1d2769bb5b03f7ca2599248079bd023f4948c9b26a06125f582625d71d5af5c Oct 31 00:44:57.507411 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:57.507373 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" event={"ID":"b76867ad-cef7-4d05-a375-a2e01134301d","Type":"ContainerStarted","Data":"c16663d9fca32d5f1c6aae2d0c5b95878e285f477701103ae3ba88388daacafe"} Oct 31 00:44:57.507612 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:57.507479 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" event={"ID":"b76867ad-cef7-4d05-a375-a2e01134301d","Type":"ContainerStarted","Data":"f1d2769bb5b03f7ca2599248079bd023f4948c9b26a06125f582625d71d5af5c"} Oct 31 00:44:57.653464 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:44:57.653424 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="278566d1-57a7-40eb-8922-91c4379e420b" path="/var/lib/kubelet/pods/278566d1-57a7-40eb-8922-91c4379e420b/volumes" Oct 31 00:45:00.508739 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:45:00.508644 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:45:01.034209 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:01.034170 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" Oct 31 00:45:01.039600 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:01.039560 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused" Oct 31 00:45:01.039959 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:01.039938 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:45:03.533997 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:03.533971 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv_b76867ad-cef7-4d05-a375-a2e01134301d/storage-initializer/0.log" Oct 31 00:45:03.534318 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:03.534009 5939 generic.go:356] "Generic (PLEG): container finished" podID="b76867ad-cef7-4d05-a375-a2e01134301d" containerID="c16663d9fca32d5f1c6aae2d0c5b95878e285f477701103ae3ba88388daacafe" exitCode=1 Oct 31 00:45:03.534318 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:03.534038 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" event={"ID":"b76867ad-cef7-4d05-a375-a2e01134301d","Type":"ContainerDied","Data":"c16663d9fca32d5f1c6aae2d0c5b95878e285f477701103ae3ba88388daacafe"} Oct 31 00:45:04.540640 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:04.540612 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv_b76867ad-cef7-4d05-a375-a2e01134301d/storage-initializer/0.log" Oct 31 00:45:04.541047 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:04.540682 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" event={"ID":"b76867ad-cef7-4d05-a375-a2e01134301d","Type":"ContainerStarted","Data":"316ccd76a382324840278f059a1bbdde7ac8bfab87a83e345394eb2a3ce73e24"} Oct 31 00:45:06.033653 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:06.033614 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" Oct 31 00:45:06.471215 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:06.471186 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv"] Oct 31 00:45:06.471430 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:06.471388 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" podUID="b76867ad-cef7-4d05-a375-a2e01134301d" containerName="storage-initializer" containerID="cri-o://316ccd76a382324840278f059a1bbdde7ac8bfab87a83e345394eb2a3ce73e24" gracePeriod=30 Oct 31 00:45:07.309561 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.309526 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv_b76867ad-cef7-4d05-a375-a2e01134301d/storage-initializer/1.log" Oct 31 00:45:07.309915 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.309902 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv_b76867ad-cef7-4d05-a375-a2e01134301d/storage-initializer/0.log" Oct 31 00:45:07.312152 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.312137 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:45:07.406426 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.406402 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b76867ad-cef7-4d05-a375-a2e01134301d-kube-api-access-xfrtv\" (UniqueName: \"kubernetes.io/projected/b76867ad-cef7-4d05-a375-a2e01134301d-kube-api-access-xfrtv\") pod \"b76867ad-cef7-4d05-a375-a2e01134301d\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " Oct 31 00:45:07.406551 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.406444 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b76867ad-cef7-4d05-a375-a2e01134301d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b76867ad-cef7-4d05-a375-a2e01134301d-kserve-provision-location\") pod \"b76867ad-cef7-4d05-a375-a2e01134301d\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " Oct 31 00:45:07.406551 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.406467 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b76867ad-cef7-4d05-a375-a2e01134301d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b76867ad-cef7-4d05-a375-a2e01134301d-cabundle-cert\") pod \"b76867ad-cef7-4d05-a375-a2e01134301d\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " Oct 31 00:45:07.406551 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.406509 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b76867ad-cef7-4d05-a375-a2e01134301d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b76867ad-cef7-4d05-a375-a2e01134301d-kube-rbac-proxy-sar-config\") pod \"b76867ad-cef7-4d05-a375-a2e01134301d\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " Oct 31 00:45:07.406551 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.406538 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b76867ad-cef7-4d05-a375-a2e01134301d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/b76867ad-cef7-4d05-a375-a2e01134301d-proxy-tls\") pod \"b76867ad-cef7-4d05-a375-a2e01134301d\" (UID: \"b76867ad-cef7-4d05-a375-a2e01134301d\") " Oct 31 00:45:07.406805 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.406714 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76867ad-cef7-4d05-a375-a2e01134301d-kserve-provision-location" pod "b76867ad-cef7-4d05-a375-a2e01134301d" (UID: "b76867ad-cef7-4d05-a375-a2e01134301d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:07.406880 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.406860 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76867ad-cef7-4d05-a375-a2e01134301d-kube-rbac-proxy-sar-config" pod "b76867ad-cef7-4d05-a375-a2e01134301d" (UID: "b76867ad-cef7-4d05-a375-a2e01134301d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:07.406912 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.406856 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76867ad-cef7-4d05-a375-a2e01134301d-cabundle-cert" pod "b76867ad-cef7-4d05-a375-a2e01134301d" (UID: "b76867ad-cef7-4d05-a375-a2e01134301d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:07.408466 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.408442 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b76867ad-cef7-4d05-a375-a2e01134301d-proxy-tls" pod "b76867ad-cef7-4d05-a375-a2e01134301d" (UID: "b76867ad-cef7-4d05-a375-a2e01134301d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:07.408580 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.408561 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76867ad-cef7-4d05-a375-a2e01134301d-kube-api-access-xfrtv" pod "b76867ad-cef7-4d05-a375-a2e01134301d" (UID: "b76867ad-cef7-4d05-a375-a2e01134301d"). InnerVolumeSpecName "kube-api-access-xfrtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:07.507161 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.507132 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b76867ad-cef7-4d05-a375-a2e01134301d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:07.507161 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.507158 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b76867ad-cef7-4d05-a375-a2e01134301d-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:07.507328 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.507173 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfrtv\" (UniqueName: \"kubernetes.io/projected/b76867ad-cef7-4d05-a375-a2e01134301d-kube-api-access-xfrtv\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:07.507328 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.507185 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b76867ad-cef7-4d05-a375-a2e01134301d-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:07.507328 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.507193 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b76867ad-cef7-4d05-a375-a2e01134301d-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:07.556355 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.556332 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv_b76867ad-cef7-4d05-a375-a2e01134301d/storage-initializer/1.log" Oct 31 00:45:07.556750 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.556732 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv_b76867ad-cef7-4d05-a375-a2e01134301d/storage-initializer/0.log" Oct 31 00:45:07.556850 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.556771 5939 generic.go:356] "Generic (PLEG): container finished" podID="b76867ad-cef7-4d05-a375-a2e01134301d" containerID="316ccd76a382324840278f059a1bbdde7ac8bfab87a83e345394eb2a3ce73e24" exitCode=1 Oct 31 00:45:07.556850 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.556823 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" Oct 31 00:45:07.556923 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.556850 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" event={"ID":"b76867ad-cef7-4d05-a375-a2e01134301d","Type":"ContainerDied","Data":"316ccd76a382324840278f059a1bbdde7ac8bfab87a83e345394eb2a3ce73e24"} Oct 31 00:45:07.556923 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.556914 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv" event={"ID":"b76867ad-cef7-4d05-a375-a2e01134301d","Type":"ContainerDied","Data":"f1d2769bb5b03f7ca2599248079bd023f4948c9b26a06125f582625d71d5af5c"} Oct 31 00:45:07.556985 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.556940 5939 scope.go:122] "RemoveContainer" containerID="316ccd76a382324840278f059a1bbdde7ac8bfab87a83e345394eb2a3ce73e24" Oct 31 00:45:07.565950 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.565934 5939 scope.go:122] "RemoveContainer" containerID="c16663d9fca32d5f1c6aae2d0c5b95878e285f477701103ae3ba88388daacafe" Oct 31 00:45:07.573240 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.573225 5939 scope.go:122] "RemoveContainer" containerID="316ccd76a382324840278f059a1bbdde7ac8bfab87a83e345394eb2a3ce73e24" Oct 31 00:45:07.573469 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:45:07.573451 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316ccd76a382324840278f059a1bbdde7ac8bfab87a83e345394eb2a3ce73e24\": container with ID starting with 316ccd76a382324840278f059a1bbdde7ac8bfab87a83e345394eb2a3ce73e24 not found: ID does not exist" containerID="316ccd76a382324840278f059a1bbdde7ac8bfab87a83e345394eb2a3ce73e24" Oct 31 00:45:07.573520 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.573477 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316ccd76a382324840278f059a1bbdde7ac8bfab87a83e345394eb2a3ce73e24"} err="failed to get container status \"316ccd76a382324840278f059a1bbdde7ac8bfab87a83e345394eb2a3ce73e24\": rpc error: code = NotFound desc = could not find container \"316ccd76a382324840278f059a1bbdde7ac8bfab87a83e345394eb2a3ce73e24\": container with ID starting with 316ccd76a382324840278f059a1bbdde7ac8bfab87a83e345394eb2a3ce73e24 not found: ID does not exist" Oct 31 00:45:07.573520 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.573491 5939 scope.go:122] "RemoveContainer" containerID="c16663d9fca32d5f1c6aae2d0c5b95878e285f477701103ae3ba88388daacafe" Oct 31 00:45:07.573781 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:45:07.573762 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c16663d9fca32d5f1c6aae2d0c5b95878e285f477701103ae3ba88388daacafe\": container with ID starting with c16663d9fca32d5f1c6aae2d0c5b95878e285f477701103ae3ba88388daacafe not found: ID does not exist" containerID="c16663d9fca32d5f1c6aae2d0c5b95878e285f477701103ae3ba88388daacafe" Oct 31 00:45:07.573832 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.573786 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c16663d9fca32d5f1c6aae2d0c5b95878e285f477701103ae3ba88388daacafe"} err="failed to get container status \"c16663d9fca32d5f1c6aae2d0c5b95878e285f477701103ae3ba88388daacafe\": rpc error: code = NotFound desc = could not find container \"c16663d9fca32d5f1c6aae2d0c5b95878e285f477701103ae3ba88388daacafe\": container with ID starting with c16663d9fca32d5f1c6aae2d0c5b95878e285f477701103ae3ba88388daacafe not found: ID does not exist" Oct 31 00:45:07.601519 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.601499 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv"] Oct 31 00:45:07.607473 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.607446 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-init-fail-5e43f7-predictor-7478b4df9-nm5qv"] Oct 31 00:45:07.651686 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:07.651642 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b76867ad-cef7-4d05-a375-a2e01134301d" path="/var/lib/kubelet/pods/b76867ad-cef7-4d05-a375-a2e01134301d/volumes" Oct 31 00:45:09.161552 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.159630 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp"] Oct 31 00:45:09.161552 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.160514 5939 cpu_manager.go:414] "containerMap: removing container" podUID="b76867ad-cef7-4d05-a375-a2e01134301d" containerName="storage-initializer" Oct 31 00:45:09.161552 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.160533 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76867ad-cef7-4d05-a375-a2e01134301d" containerName="storage-initializer" Oct 31 00:45:09.161552 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.160566 5939 cpu_manager.go:414] "containerMap: removing container" podUID="b76867ad-cef7-4d05-a375-a2e01134301d" containerName="storage-initializer" Oct 31 00:45:09.161552 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.160574 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76867ad-cef7-4d05-a375-a2e01134301d" containerName="storage-initializer" Oct 31 00:45:09.161552 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.160802 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76867ad-cef7-4d05-a375-a2e01134301d" containerName="storage-initializer" Oct 31 00:45:09.161552 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.160824 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76867ad-cef7-4d05-a375-a2e01134301d" containerName="storage-initializer" Oct 31 00:45:09.167782 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.167752 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:09.175953 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.175761 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:45:09.175953 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.175810 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 00:45:09.175953 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.175844 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-predictive-sklearn-predictor-serving-cert\"" Oct 31 00:45:09.175953 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.175863 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 00:45:09.175953 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.175895 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 00:45:09.176511 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.176054 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\"" Oct 31 00:45:09.176511 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.176175 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 00:45:09.177864 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.177839 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp"] Oct 31 00:45:09.218370 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.218343 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:09.218515 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.218385 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:09.218515 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.218470 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:09.218614 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.218514 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:09.218614 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.218546 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvm7c\" (UniqueName: \"kubernetes.io/projected/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kube-api-access-mvm7c\") pod \"isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:09.319584 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.319553 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:09.319757 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.319590 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:09.319757 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.319612 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvm7c\" (UniqueName: \"kubernetes.io/projected/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kube-api-access-mvm7c\") pod \"isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:09.319757 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.319663 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:09.319757 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.319721 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:09.320012 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.319993 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:09.320415 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.320393 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:09.320482 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.320468 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:09.322106 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.322084 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:09.334000 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.333979 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvm7c\" (UniqueName: \"kubernetes.io/projected/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kube-api-access-mvm7c\") pod \"isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:09.488371 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.488280 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:09.769483 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:09.769465 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp"] Oct 31 00:45:09.774873 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:45:09.774847 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0c3988_460d_4ef5_a00f_3bf7d9d2a4f7.slice/crio-632e8b53b0d468b90202ba870751882c69b4ab674b97019aace9b78b20618b05 WatchSource:0}: Error finding container 632e8b53b0d468b90202ba870751882c69b4ab674b97019aace9b78b20618b05: Status 404 returned error can't find the container with id 632e8b53b0d468b90202ba870751882c69b4ab674b97019aace9b78b20618b05 Oct 31 00:45:10.574721 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:10.574686 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" event={"ID":"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7","Type":"ContainerStarted","Data":"3eea5bd9aa4fd14344a30376c72cfd08abe3f1177808fea4e23f5cc15de29049"} Oct 31 00:45:10.575123 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:10.574741 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" event={"ID":"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7","Type":"ContainerStarted","Data":"632e8b53b0d468b90202ba870751882c69b4ab674b97019aace9b78b20618b05"} Oct 31 00:45:11.033273 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:11.033236 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" Oct 31 00:45:11.038752 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:11.038720 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused" Oct 31 00:45:11.038904 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:11.038815 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:45:11.039147 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:11.039122 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:45:11.039229 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:11.039219 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:45:13.359249 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.359224 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:45:13.454479 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.454444 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4721086b-ea72-46e3-8569-cbc34d39fa0e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4721086b-ea72-46e3-8569-cbc34d39fa0e-kube-rbac-proxy-sar-config\") pod \"4721086b-ea72-46e3-8569-cbc34d39fa0e\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " Oct 31 00:45:13.454709 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.454496 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4721086b-ea72-46e3-8569-cbc34d39fa0e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4721086b-ea72-46e3-8569-cbc34d39fa0e-kserve-provision-location\") pod \"4721086b-ea72-46e3-8569-cbc34d39fa0e\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " Oct 31 00:45:13.454709 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.454546 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4721086b-ea72-46e3-8569-cbc34d39fa0e-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4721086b-ea72-46e3-8569-cbc34d39fa0e-cabundle-cert\") pod \"4721086b-ea72-46e3-8569-cbc34d39fa0e\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " Oct 31 00:45:13.454709 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.454606 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4721086b-ea72-46e3-8569-cbc34d39fa0e-kube-api-access-zpcgv\" (UniqueName: \"kubernetes.io/projected/4721086b-ea72-46e3-8569-cbc34d39fa0e-kube-api-access-zpcgv\") pod \"4721086b-ea72-46e3-8569-cbc34d39fa0e\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " Oct 31 00:45:13.454709 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.454683 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4721086b-ea72-46e3-8569-cbc34d39fa0e-proxy-tls\" (UniqueName: \"kubernetes.io/secret/4721086b-ea72-46e3-8569-cbc34d39fa0e-proxy-tls\") pod \"4721086b-ea72-46e3-8569-cbc34d39fa0e\" (UID: \"4721086b-ea72-46e3-8569-cbc34d39fa0e\") " Oct 31 00:45:13.454935 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.454854 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4721086b-ea72-46e3-8569-cbc34d39fa0e-kube-rbac-proxy-sar-config" pod "4721086b-ea72-46e3-8569-cbc34d39fa0e" (UID: "4721086b-ea72-46e3-8569-cbc34d39fa0e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:13.454982 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.454882 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4721086b-ea72-46e3-8569-cbc34d39fa0e-kserve-provision-location" pod "4721086b-ea72-46e3-8569-cbc34d39fa0e" (UID: "4721086b-ea72-46e3-8569-cbc34d39fa0e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:13.455050 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.455029 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4721086b-ea72-46e3-8569-cbc34d39fa0e-cabundle-cert" pod "4721086b-ea72-46e3-8569-cbc34d39fa0e" (UID: "4721086b-ea72-46e3-8569-cbc34d39fa0e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:13.456784 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.456760 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4721086b-ea72-46e3-8569-cbc34d39fa0e-proxy-tls" pod "4721086b-ea72-46e3-8569-cbc34d39fa0e" (UID: "4721086b-ea72-46e3-8569-cbc34d39fa0e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:13.456889 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.456786 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4721086b-ea72-46e3-8569-cbc34d39fa0e-kube-api-access-zpcgv" pod "4721086b-ea72-46e3-8569-cbc34d39fa0e" (UID: "4721086b-ea72-46e3-8569-cbc34d39fa0e"). InnerVolumeSpecName "kube-api-access-zpcgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:13.555819 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.555795 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4721086b-ea72-46e3-8569-cbc34d39fa0e-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:13.555819 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.555820 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4721086b-ea72-46e3-8569-cbc34d39fa0e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:13.555955 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.555831 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4721086b-ea72-46e3-8569-cbc34d39fa0e-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:13.555955 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.555840 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4721086b-ea72-46e3-8569-cbc34d39fa0e-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:13.555955 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.555850 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpcgv\" (UniqueName: \"kubernetes.io/projected/4721086b-ea72-46e3-8569-cbc34d39fa0e-kube-api-access-zpcgv\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:13.590480 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.590453 5939 generic.go:356] "Generic (PLEG): container finished" podID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerID="3eea5bd9aa4fd14344a30376c72cfd08abe3f1177808fea4e23f5cc15de29049" exitCode=0 Oct 31 00:45:13.590574 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.590524 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" event={"ID":"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7","Type":"ContainerDied","Data":"3eea5bd9aa4fd14344a30376c72cfd08abe3f1177808fea4e23f5cc15de29049"} Oct 31 00:45:13.591873 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.591856 5939 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:45:13.592870 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.592825 5939 generic.go:356] "Generic (PLEG): container finished" podID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerID="9025c86a8a84dd674af232b841e5f7e0eb73a5672b51f8bf540378f90942afdb" exitCode=0 Oct 31 00:45:13.592928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.592888 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" Oct 31 00:45:13.592928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.592909 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" event={"ID":"4721086b-ea72-46e3-8569-cbc34d39fa0e","Type":"ContainerDied","Data":"9025c86a8a84dd674af232b841e5f7e0eb73a5672b51f8bf540378f90942afdb"} Oct 31 00:45:13.592994 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.592974 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6" event={"ID":"4721086b-ea72-46e3-8569-cbc34d39fa0e","Type":"ContainerDied","Data":"34bab6fb53cd99f2570191f0997d7cec113bf39cae742bbcfeee0af2608c831b"} Oct 31 00:45:13.593025 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.593005 5939 scope.go:122] "RemoveContainer" containerID="9025c86a8a84dd674af232b841e5f7e0eb73a5672b51f8bf540378f90942afdb" Oct 31 00:45:13.607072 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.607055 5939 scope.go:122] "RemoveContainer" containerID="9e1dd6bb1cd2977367a832bbe7a3eafba1e99bac4583e896ecd7f98fcdf01fbb" Oct 31 00:45:13.614207 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.614192 5939 scope.go:122] "RemoveContainer" containerID="9b456590b6ae367ac21cb7b4dfaf400ef369728185fbbcd57abf8d0aa41956a6" Oct 31 00:45:13.621443 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.621424 5939 scope.go:122] "RemoveContainer" containerID="dbb32854b6020b6bbffe5699625893b19e899a26e4aa169225d143caba3ef3d0" Oct 31 00:45:13.628231 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.628215 5939 scope.go:122] "RemoveContainer" containerID="9025c86a8a84dd674af232b841e5f7e0eb73a5672b51f8bf540378f90942afdb" Oct 31 00:45:13.628522 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:45:13.628500 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9025c86a8a84dd674af232b841e5f7e0eb73a5672b51f8bf540378f90942afdb\": container with ID starting with 9025c86a8a84dd674af232b841e5f7e0eb73a5672b51f8bf540378f90942afdb not found: ID does not exist" containerID="9025c86a8a84dd674af232b841e5f7e0eb73a5672b51f8bf540378f90942afdb" Oct 31 00:45:13.628610 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.628528 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9025c86a8a84dd674af232b841e5f7e0eb73a5672b51f8bf540378f90942afdb"} err="failed to get container status \"9025c86a8a84dd674af232b841e5f7e0eb73a5672b51f8bf540378f90942afdb\": rpc error: code = NotFound desc = could not find container \"9025c86a8a84dd674af232b841e5f7e0eb73a5672b51f8bf540378f90942afdb\": container with ID starting with 9025c86a8a84dd674af232b841e5f7e0eb73a5672b51f8bf540378f90942afdb not found: ID does not exist" Oct 31 00:45:13.628610 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.628544 5939 scope.go:122] "RemoveContainer" containerID="9e1dd6bb1cd2977367a832bbe7a3eafba1e99bac4583e896ecd7f98fcdf01fbb" Oct 31 00:45:13.628869 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:45:13.628852 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e1dd6bb1cd2977367a832bbe7a3eafba1e99bac4583e896ecd7f98fcdf01fbb\": container with ID starting with 9e1dd6bb1cd2977367a832bbe7a3eafba1e99bac4583e896ecd7f98fcdf01fbb not found: ID does not exist" containerID="9e1dd6bb1cd2977367a832bbe7a3eafba1e99bac4583e896ecd7f98fcdf01fbb" Oct 31 00:45:13.628913 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.628875 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e1dd6bb1cd2977367a832bbe7a3eafba1e99bac4583e896ecd7f98fcdf01fbb"} err="failed to get container status \"9e1dd6bb1cd2977367a832bbe7a3eafba1e99bac4583e896ecd7f98fcdf01fbb\": rpc error: code = NotFound desc = could not find container \"9e1dd6bb1cd2977367a832bbe7a3eafba1e99bac4583e896ecd7f98fcdf01fbb\": container with ID starting with 9e1dd6bb1cd2977367a832bbe7a3eafba1e99bac4583e896ecd7f98fcdf01fbb not found: ID does not exist" Oct 31 00:45:13.628913 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.628889 5939 scope.go:122] "RemoveContainer" containerID="9b456590b6ae367ac21cb7b4dfaf400ef369728185fbbcd57abf8d0aa41956a6" Oct 31 00:45:13.629136 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:45:13.629118 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b456590b6ae367ac21cb7b4dfaf400ef369728185fbbcd57abf8d0aa41956a6\": container with ID starting with 9b456590b6ae367ac21cb7b4dfaf400ef369728185fbbcd57abf8d0aa41956a6 not found: ID does not exist" containerID="9b456590b6ae367ac21cb7b4dfaf400ef369728185fbbcd57abf8d0aa41956a6" Oct 31 00:45:13.629184 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.629142 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b456590b6ae367ac21cb7b4dfaf400ef369728185fbbcd57abf8d0aa41956a6"} err="failed to get container status \"9b456590b6ae367ac21cb7b4dfaf400ef369728185fbbcd57abf8d0aa41956a6\": rpc error: code = NotFound desc = could not find container \"9b456590b6ae367ac21cb7b4dfaf400ef369728185fbbcd57abf8d0aa41956a6\": container with ID starting with 9b456590b6ae367ac21cb7b4dfaf400ef369728185fbbcd57abf8d0aa41956a6 not found: ID does not exist" Oct 31 00:45:13.629184 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.629156 5939 scope.go:122] "RemoveContainer" containerID="dbb32854b6020b6bbffe5699625893b19e899a26e4aa169225d143caba3ef3d0" Oct 31 00:45:13.629395 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:45:13.629378 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb32854b6020b6bbffe5699625893b19e899a26e4aa169225d143caba3ef3d0\": container with ID starting with dbb32854b6020b6bbffe5699625893b19e899a26e4aa169225d143caba3ef3d0 not found: ID does not exist" containerID="dbb32854b6020b6bbffe5699625893b19e899a26e4aa169225d143caba3ef3d0" Oct 31 00:45:13.629447 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.629403 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb32854b6020b6bbffe5699625893b19e899a26e4aa169225d143caba3ef3d0"} err="failed to get container status \"dbb32854b6020b6bbffe5699625893b19e899a26e4aa169225d143caba3ef3d0\": rpc error: code = NotFound desc = could not find container \"dbb32854b6020b6bbffe5699625893b19e899a26e4aa169225d143caba3ef3d0\": container with ID starting with dbb32854b6020b6bbffe5699625893b19e899a26e4aa169225d143caba3ef3d0 not found: ID does not exist" Oct 31 00:45:13.633839 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.633819 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6"] Oct 31 00:45:13.640616 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.640590 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-batcher-custom-predictor-7665b8df8d-xfxk6"] Oct 31 00:45:13.651815 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:13.651793 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" path="/var/lib/kubelet/pods/4721086b-ea72-46e3-8569-cbc34d39fa0e/volumes" Oct 31 00:45:30.515146 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:45:30.515107 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:45:35.723847 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:35.723813 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" event={"ID":"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7","Type":"ContainerStarted","Data":"d6fcaf1edcb87bfb6254aad6115738c14a8b477fc8cf32e3e8e477d3a6637716"} Oct 31 00:45:35.724286 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:35.723873 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" event={"ID":"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7","Type":"ContainerStarted","Data":"f603220285df0b911c5ee66869e87f792dfe57e6a3a5daf26decb32f4a4caebb"} Oct 31 00:45:35.724286 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:35.724138 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:35.724369 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:35.724283 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:35.725391 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:35.725368 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:45:35.755175 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:35.755131 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" podStartSLOduration=1.4772095140000001 podStartE2EDuration="26.755120133s" totalImagesPullingTime="21.568818663s" totalInitContainerRuntime="3.709091956s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:35.753388518 +0000 UTC m=+948.242663696" watchObservedRunningTime="2026-10-31 00:45:35.755120133 +0000 UTC m=+948.244395311" Oct 31 00:45:36.728995 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:36.728955 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:45:41.733646 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:41.733616 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:45:41.734307 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:41.734275 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:45:51.734722 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:45:51.734654 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:46:00.508382 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:46:00.508342 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:46:01.734645 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:01.734607 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:46:11.735131 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:11.735083 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:46:21.734761 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:21.734719 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:46:30.508030 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:46:30.507954 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:46:31.734419 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:31.734376 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:46:41.735326 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:41.735299 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:46:50.019230 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:50.019195 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp"] Oct 31 00:46:50.019680 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:50.019483 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kserve-container" containerID="cri-o://f603220285df0b911c5ee66869e87f792dfe57e6a3a5daf26decb32f4a4caebb" gracePeriod=30 Oct 31 00:46:50.019680 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:50.019506 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kube-rbac-proxy" containerID="cri-o://d6fcaf1edcb87bfb6254aad6115738c14a8b477fc8cf32e3e8e477d3a6637716" gracePeriod=30 Oct 31 00:46:51.027991 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:51.027953 5939 generic.go:356] "Generic (PLEG): container finished" podID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerID="d6fcaf1edcb87bfb6254aad6115738c14a8b477fc8cf32e3e8e477d3a6637716" exitCode=0 Oct 31 00:46:51.028379 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:51.028022 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" event={"ID":"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7","Type":"ContainerDied","Data":"d6fcaf1edcb87bfb6254aad6115738c14a8b477fc8cf32e3e8e477d3a6637716"} Oct 31 00:46:51.729938 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:51.729891 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused" Oct 31 00:46:51.734436 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:51.734409 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 31 00:46:54.668128 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:54.668104 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:46:54.731503 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:54.731473 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-proxy-tls\") pod \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " Oct 31 00:46:54.731699 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:54.731527 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kube-rbac-proxy-sar-config\") pod \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " Oct 31 00:46:54.731699 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:54.731545 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kube-api-access-mvm7c\" (UniqueName: \"kubernetes.io/projected/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kube-api-access-mvm7c\") pod \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " Oct 31 00:46:54.731699 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:54.731593 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kserve-provision-location\") pod \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " Oct 31 00:46:54.731699 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:54.731625 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-cabundle-cert\") pod \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\" (UID: \"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7\") " Oct 31 00:46:54.731981 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:54.731955 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kserve-provision-location" pod "cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" (UID: "cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:46:54.732042 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:54.732010 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kube-rbac-proxy-sar-config" pod "cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" (UID: "cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:54.732082 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:54.732045 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-cabundle-cert" pod "cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" (UID: "cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:54.733602 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:54.733582 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kube-api-access-mvm7c" pod "cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" (UID: "cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7"). InnerVolumeSpecName "kube-api-access-mvm7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:54.733706 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:54.733613 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-proxy-tls" pod "cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" (UID: "cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:54.832928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:54.832844 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:54.832928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:54.832873 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvm7c\" (UniqueName: \"kubernetes.io/projected/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kube-api-access-mvm7c\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:54.832928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:54.832883 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:54.832928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:54.832892 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:54.832928 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:54.832900 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:55.045569 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:55.045534 5939 generic.go:356] "Generic (PLEG): container finished" podID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerID="f603220285df0b911c5ee66869e87f792dfe57e6a3a5daf26decb32f4a4caebb" exitCode=0 Oct 31 00:46:55.045782 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:55.045595 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" Oct 31 00:46:55.045782 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:55.045621 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" event={"ID":"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7","Type":"ContainerDied","Data":"f603220285df0b911c5ee66869e87f792dfe57e6a3a5daf26decb32f4a4caebb"} Oct 31 00:46:55.045782 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:55.045702 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp" event={"ID":"cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7","Type":"ContainerDied","Data":"632e8b53b0d468b90202ba870751882c69b4ab674b97019aace9b78b20618b05"} Oct 31 00:46:55.045782 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:55.045734 5939 scope.go:122] "RemoveContainer" containerID="d6fcaf1edcb87bfb6254aad6115738c14a8b477fc8cf32e3e8e477d3a6637716" Oct 31 00:46:55.054088 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:55.054071 5939 scope.go:122] "RemoveContainer" containerID="f603220285df0b911c5ee66869e87f792dfe57e6a3a5daf26decb32f4a4caebb" Oct 31 00:46:55.061288 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:55.061269 5939 scope.go:122] "RemoveContainer" containerID="3eea5bd9aa4fd14344a30376c72cfd08abe3f1177808fea4e23f5cc15de29049" Oct 31 00:46:55.068448 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:55.068431 5939 scope.go:122] "RemoveContainer" containerID="d6fcaf1edcb87bfb6254aad6115738c14a8b477fc8cf32e3e8e477d3a6637716" Oct 31 00:46:55.068734 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:46:55.068707 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6fcaf1edcb87bfb6254aad6115738c14a8b477fc8cf32e3e8e477d3a6637716\": container with ID starting with d6fcaf1edcb87bfb6254aad6115738c14a8b477fc8cf32e3e8e477d3a6637716 not found: ID does not exist" containerID="d6fcaf1edcb87bfb6254aad6115738c14a8b477fc8cf32e3e8e477d3a6637716" Oct 31 00:46:55.068783 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:55.068733 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6fcaf1edcb87bfb6254aad6115738c14a8b477fc8cf32e3e8e477d3a6637716"} err="failed to get container status \"d6fcaf1edcb87bfb6254aad6115738c14a8b477fc8cf32e3e8e477d3a6637716\": rpc error: code = NotFound desc = could not find container \"d6fcaf1edcb87bfb6254aad6115738c14a8b477fc8cf32e3e8e477d3a6637716\": container with ID starting with d6fcaf1edcb87bfb6254aad6115738c14a8b477fc8cf32e3e8e477d3a6637716 not found: ID does not exist" Oct 31 00:46:55.068783 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:55.068749 5939 scope.go:122] "RemoveContainer" containerID="f603220285df0b911c5ee66869e87f792dfe57e6a3a5daf26decb32f4a4caebb" Oct 31 00:46:55.068974 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:46:55.068956 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f603220285df0b911c5ee66869e87f792dfe57e6a3a5daf26decb32f4a4caebb\": container with ID starting with f603220285df0b911c5ee66869e87f792dfe57e6a3a5daf26decb32f4a4caebb not found: ID does not exist" containerID="f603220285df0b911c5ee66869e87f792dfe57e6a3a5daf26decb32f4a4caebb" Oct 31 00:46:55.069019 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:55.068980 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f603220285df0b911c5ee66869e87f792dfe57e6a3a5daf26decb32f4a4caebb"} err="failed to get container status \"f603220285df0b911c5ee66869e87f792dfe57e6a3a5daf26decb32f4a4caebb\": rpc error: code = NotFound desc = could not find container \"f603220285df0b911c5ee66869e87f792dfe57e6a3a5daf26decb32f4a4caebb\": container with ID starting with f603220285df0b911c5ee66869e87f792dfe57e6a3a5daf26decb32f4a4caebb not found: ID does not exist" Oct 31 00:46:55.069019 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:55.068994 5939 scope.go:122] "RemoveContainer" containerID="3eea5bd9aa4fd14344a30376c72cfd08abe3f1177808fea4e23f5cc15de29049" Oct 31 00:46:55.069199 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:46:55.069177 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eea5bd9aa4fd14344a30376c72cfd08abe3f1177808fea4e23f5cc15de29049\": container with ID starting with 3eea5bd9aa4fd14344a30376c72cfd08abe3f1177808fea4e23f5cc15de29049 not found: ID does not exist" containerID="3eea5bd9aa4fd14344a30376c72cfd08abe3f1177808fea4e23f5cc15de29049" Oct 31 00:46:55.069238 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:55.069203 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eea5bd9aa4fd14344a30376c72cfd08abe3f1177808fea4e23f5cc15de29049"} err="failed to get container status \"3eea5bd9aa4fd14344a30376c72cfd08abe3f1177808fea4e23f5cc15de29049\": rpc error: code = NotFound desc = could not find container \"3eea5bd9aa4fd14344a30376c72cfd08abe3f1177808fea4e23f5cc15de29049\": container with ID starting with 3eea5bd9aa4fd14344a30376c72cfd08abe3f1177808fea4e23f5cc15de29049 not found: ID does not exist" Oct 31 00:46:55.072546 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:55.072525 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp"] Oct 31 00:46:55.078895 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:55.078875 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-sklearn-predictor-7c6bd47f6f-2wsxp"] Oct 31 00:46:55.651889 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:55.651860 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" path="/var/lib/kubelet/pods/cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7/volumes" Oct 31 00:46:57.508947 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.508910 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7"] Oct 31 00:46:57.509313 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509231 5939 cpu_manager.go:414] "containerMap: removing container" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kserve-container" Oct 31 00:46:57.509313 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509242 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kserve-container" Oct 31 00:46:57.509313 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509249 5939 cpu_manager.go:414] "containerMap: removing container" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kube-rbac-proxy" Oct 31 00:46:57.509313 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509254 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kube-rbac-proxy" Oct 31 00:46:57.509313 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509265 5939 cpu_manager.go:414] "containerMap: removing container" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="storage-initializer" Oct 31 00:46:57.509313 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509270 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="storage-initializer" Oct 31 00:46:57.509313 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509293 5939 cpu_manager.go:414] "containerMap: removing container" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="agent" Oct 31 00:46:57.509313 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509297 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="agent" Oct 31 00:46:57.509313 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509306 5939 cpu_manager.go:414] "containerMap: removing container" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kserve-container" Oct 31 00:46:57.509313 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509310 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kserve-container" Oct 31 00:46:57.509313 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509317 5939 cpu_manager.go:414] "containerMap: removing container" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="storage-initializer" Oct 31 00:46:57.509713 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509322 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="storage-initializer" Oct 31 00:46:57.509713 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509333 5939 cpu_manager.go:414] "containerMap: removing container" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kube-rbac-proxy" Oct 31 00:46:57.509713 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509338 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kube-rbac-proxy" Oct 31 00:46:57.509713 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509386 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kserve-container" Oct 31 00:46:57.509713 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509394 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="kube-rbac-proxy" Oct 31 00:46:57.509713 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509405 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4721086b-ea72-46e3-8569-cbc34d39fa0e" containerName="agent" Oct 31 00:46:57.509713 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509411 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kserve-container" Oct 31 00:46:57.509713 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.509417 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0c3988-460d-4ef5-a00f-3bf7d9d2a4f7" containerName="kube-rbac-proxy" Oct 31 00:46:57.512162 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.512143 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:46:57.518914 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.518897 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 00:46:57.519655 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.519638 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 00:46:57.519655 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.519652 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\"" Oct 31 00:46:57.519821 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.519785 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:46:57.520122 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.520107 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-predictive-xgboost-predictor-serving-cert\"" Oct 31 00:46:57.520122 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.520109 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 00:46:57.520260 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.520236 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 00:46:57.526899 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.526705 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7"] Oct 31 00:46:57.556321 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.556292 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqvdx\" (UniqueName: \"kubernetes.io/projected/f85f3189-0769-4053-b252-c48e9b9afe31-kube-api-access-wqvdx\") pod \"isvc-predictive-xgboost-predictor-cd7fb98bb-668x7\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:46:57.556321 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.556324 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f85f3189-0769-4053-b252-c48e9b9afe31-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-cd7fb98bb-668x7\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:46:57.556506 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.556350 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f85f3189-0769-4053-b252-c48e9b9afe31-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-cd7fb98bb-668x7\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:46:57.556506 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.556456 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f85f3189-0769-4053-b252-c48e9b9afe31-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-cd7fb98bb-668x7\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:46:57.556582 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.556510 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f85f3189-0769-4053-b252-c48e9b9afe31-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-cd7fb98bb-668x7\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:46:57.657796 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.657764 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f85f3189-0769-4053-b252-c48e9b9afe31-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-cd7fb98bb-668x7\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:46:57.657989 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.657821 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f85f3189-0769-4053-b252-c48e9b9afe31-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-cd7fb98bb-668x7\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:46:57.657989 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.657862 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqvdx\" (UniqueName: \"kubernetes.io/projected/f85f3189-0769-4053-b252-c48e9b9afe31-kube-api-access-wqvdx\") pod \"isvc-predictive-xgboost-predictor-cd7fb98bb-668x7\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:46:57.657989 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.657887 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f85f3189-0769-4053-b252-c48e9b9afe31-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-cd7fb98bb-668x7\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:46:57.657989 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.657921 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f85f3189-0769-4053-b252-c48e9b9afe31-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-cd7fb98bb-668x7\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:46:57.658309 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.658287 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f85f3189-0769-4053-b252-c48e9b9afe31-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-cd7fb98bb-668x7\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:46:57.658441 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.658417 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f85f3189-0769-4053-b252-c48e9b9afe31-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-cd7fb98bb-668x7\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:46:57.658528 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.658512 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f85f3189-0769-4053-b252-c48e9b9afe31-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-cd7fb98bb-668x7\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:46:57.660321 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.660305 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f85f3189-0769-4053-b252-c48e9b9afe31-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-cd7fb98bb-668x7\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:46:57.673353 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.673322 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqvdx\" (UniqueName: \"kubernetes.io/projected/f85f3189-0769-4053-b252-c48e9b9afe31-kube-api-access-wqvdx\") pod \"isvc-predictive-xgboost-predictor-cd7fb98bb-668x7\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:46:57.826070 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:57.825978 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:46:58.108553 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:58.108532 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7"] Oct 31 00:46:58.110449 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:46:58.110420 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85f3189_0769_4053_b252_c48e9b9afe31.slice/crio-2c09364cad08ae5cba0843e8ae6dbd15878251fcd6f07b15d88e225dafe9e6bf WatchSource:0}: Error finding container 2c09364cad08ae5cba0843e8ae6dbd15878251fcd6f07b15d88e225dafe9e6bf: Status 404 returned error can't find the container with id 2c09364cad08ae5cba0843e8ae6dbd15878251fcd6f07b15d88e225dafe9e6bf Oct 31 00:46:59.062884 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:59.062843 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" event={"ID":"f85f3189-0769-4053-b252-c48e9b9afe31","Type":"ContainerStarted","Data":"01a17b5bcade8c364e93e32517887055b3fb004cd6879c08f72ed2642a0b64f4"} Oct 31 00:46:59.063262 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:46:59.062915 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" event={"ID":"f85f3189-0769-4053-b252-c48e9b9afe31","Type":"ContainerStarted","Data":"2c09364cad08ae5cba0843e8ae6dbd15878251fcd6f07b15d88e225dafe9e6bf"} Oct 31 00:47:00.508275 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:47:00.508234 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:47:02.076680 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:47:02.076630 5939 generic.go:356] "Generic (PLEG): container finished" podID="f85f3189-0769-4053-b252-c48e9b9afe31" containerID="01a17b5bcade8c364e93e32517887055b3fb004cd6879c08f72ed2642a0b64f4" exitCode=0 Oct 31 00:47:02.077059 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:47:02.076708 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" event={"ID":"f85f3189-0769-4053-b252-c48e9b9afe31","Type":"ContainerDied","Data":"01a17b5bcade8c364e93e32517887055b3fb004cd6879c08f72ed2642a0b64f4"} Oct 31 00:47:03.083529 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:47:03.083497 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" event={"ID":"f85f3189-0769-4053-b252-c48e9b9afe31","Type":"ContainerStarted","Data":"1a3e2989d855ad99a91c6699dd9b003034fb62e249916034bdc1e5177efa9891"} Oct 31 00:47:03.083933 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:47:03.083557 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" event={"ID":"f85f3189-0769-4053-b252-c48e9b9afe31","Type":"ContainerStarted","Data":"e94b110fe34d708177be646e259dff2f9952f65c75b4628a5e4bff2dd0c8a9e2"} Oct 31 00:47:03.083933 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:47:03.083741 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:47:03.114032 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:47:03.113959 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" podStartSLOduration=2.377617984 podStartE2EDuration="6.113945737s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.736327753s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:03.111600811 +0000 UTC m=+1035.600875991" watchObservedRunningTime="2026-10-31 00:47:03.113945737 +0000 UTC m=+1035.603220921" Oct 31 00:47:04.087550 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:47:04.087516 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:47:04.088698 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:47:04.088651 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 31 00:47:05.091968 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:47:05.091920 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 31 00:47:10.096641 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:47:10.096610 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:47:10.097167 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:47:10.097138 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 31 00:47:20.097469 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:47:20.097425 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 31 00:47:30.097458 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:47:30.097417 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 31 00:47:30.509149 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:47:30.509109 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:47:40.097330 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:47:40.097294 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 31 00:47:50.097071 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:47:50.097025 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 31 00:48:00.097334 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:00.097246 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 31 00:48:00.509521 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:48:00.509478 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:48:10.098437 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:10.098406 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:48:18.312130 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:18.312097 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7"] Oct 31 00:48:18.312782 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:18.312330 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kserve-container" containerID="cri-o://e94b110fe34d708177be646e259dff2f9952f65c75b4628a5e4bff2dd0c8a9e2" gracePeriod=30 Oct 31 00:48:18.312782 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:18.312378 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kube-rbac-proxy" containerID="cri-o://1a3e2989d855ad99a91c6699dd9b003034fb62e249916034bdc1e5177efa9891" gracePeriod=30 Oct 31 00:48:19.398910 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:19.398875 5939 generic.go:356] "Generic (PLEG): container finished" podID="f85f3189-0769-4053-b252-c48e9b9afe31" containerID="1a3e2989d855ad99a91c6699dd9b003034fb62e249916034bdc1e5177efa9891" exitCode=0 Oct 31 00:48:19.399303 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:19.398941 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" event={"ID":"f85f3189-0769-4053-b252-c48e9b9afe31","Type":"ContainerDied","Data":"1a3e2989d855ad99a91c6699dd9b003034fb62e249916034bdc1e5177efa9891"} Oct 31 00:48:20.093064 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:20.093020 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused" Oct 31 00:48:20.097374 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:20.097351 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 31 00:48:22.952412 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:22.952385 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:48:23.079870 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.079779 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f85f3189-0769-4053-b252-c48e9b9afe31-proxy-tls\" (UniqueName: \"kubernetes.io/secret/f85f3189-0769-4053-b252-c48e9b9afe31-proxy-tls\") pod \"f85f3189-0769-4053-b252-c48e9b9afe31\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " Oct 31 00:48:23.079870 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.079840 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f85f3189-0769-4053-b252-c48e9b9afe31-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f85f3189-0769-4053-b252-c48e9b9afe31-kube-rbac-proxy-sar-config\") pod \"f85f3189-0769-4053-b252-c48e9b9afe31\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " Oct 31 00:48:23.080115 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.079897 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85f3189-0769-4053-b252-c48e9b9afe31-kube-api-access-wqvdx\" (UniqueName: \"kubernetes.io/projected/f85f3189-0769-4053-b252-c48e9b9afe31-kube-api-access-wqvdx\") pod \"f85f3189-0769-4053-b252-c48e9b9afe31\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " Oct 31 00:48:23.080115 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.079923 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f85f3189-0769-4053-b252-c48e9b9afe31-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f85f3189-0769-4053-b252-c48e9b9afe31-cabundle-cert\") pod \"f85f3189-0769-4053-b252-c48e9b9afe31\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " Oct 31 00:48:23.080115 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.079982 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85f3189-0769-4053-b252-c48e9b9afe31-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f85f3189-0769-4053-b252-c48e9b9afe31-kserve-provision-location\") pod \"f85f3189-0769-4053-b252-c48e9b9afe31\" (UID: \"f85f3189-0769-4053-b252-c48e9b9afe31\") " Oct 31 00:48:23.080423 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.080392 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f85f3189-0769-4053-b252-c48e9b9afe31-kube-rbac-proxy-sar-config" pod "f85f3189-0769-4053-b252-c48e9b9afe31" (UID: "f85f3189-0769-4053-b252-c48e9b9afe31"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:23.080508 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.080426 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85f3189-0769-4053-b252-c48e9b9afe31-kserve-provision-location" pod "f85f3189-0769-4053-b252-c48e9b9afe31" (UID: "f85f3189-0769-4053-b252-c48e9b9afe31"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:23.080508 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.080430 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f85f3189-0769-4053-b252-c48e9b9afe31-cabundle-cert" pod "f85f3189-0769-4053-b252-c48e9b9afe31" (UID: "f85f3189-0769-4053-b252-c48e9b9afe31"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:23.082043 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.082020 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f85f3189-0769-4053-b252-c48e9b9afe31-proxy-tls" pod "f85f3189-0769-4053-b252-c48e9b9afe31" (UID: "f85f3189-0769-4053-b252-c48e9b9afe31"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:23.082149 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.082057 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85f3189-0769-4053-b252-c48e9b9afe31-kube-api-access-wqvdx" pod "f85f3189-0769-4053-b252-c48e9b9afe31" (UID: "f85f3189-0769-4053-b252-c48e9b9afe31"). InnerVolumeSpecName "kube-api-access-wqvdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:23.180821 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.180787 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f85f3189-0769-4053-b252-c48e9b9afe31-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:23.180821 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.180814 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqvdx\" (UniqueName: \"kubernetes.io/projected/f85f3189-0769-4053-b252-c48e9b9afe31-kube-api-access-wqvdx\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:23.180821 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.180824 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f85f3189-0769-4053-b252-c48e9b9afe31-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:23.180821 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.180833 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f85f3189-0769-4053-b252-c48e9b9afe31-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:23.181086 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.180844 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f85f3189-0769-4053-b252-c48e9b9afe31-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:23.418686 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.418629 5939 generic.go:356] "Generic (PLEG): container finished" podID="f85f3189-0769-4053-b252-c48e9b9afe31" containerID="e94b110fe34d708177be646e259dff2f9952f65c75b4628a5e4bff2dd0c8a9e2" exitCode=0 Oct 31 00:48:23.418887 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.418691 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" event={"ID":"f85f3189-0769-4053-b252-c48e9b9afe31","Type":"ContainerDied","Data":"e94b110fe34d708177be646e259dff2f9952f65c75b4628a5e4bff2dd0c8a9e2"} Oct 31 00:48:23.418887 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.418707 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" Oct 31 00:48:23.418887 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.418739 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7" event={"ID":"f85f3189-0769-4053-b252-c48e9b9afe31","Type":"ContainerDied","Data":"2c09364cad08ae5cba0843e8ae6dbd15878251fcd6f07b15d88e225dafe9e6bf"} Oct 31 00:48:23.418887 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.418774 5939 scope.go:122] "RemoveContainer" containerID="1a3e2989d855ad99a91c6699dd9b003034fb62e249916034bdc1e5177efa9891" Oct 31 00:48:23.427964 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.427945 5939 scope.go:122] "RemoveContainer" containerID="e94b110fe34d708177be646e259dff2f9952f65c75b4628a5e4bff2dd0c8a9e2" Oct 31 00:48:23.435880 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.435862 5939 scope.go:122] "RemoveContainer" containerID="01a17b5bcade8c364e93e32517887055b3fb004cd6879c08f72ed2642a0b64f4" Oct 31 00:48:23.443490 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.443470 5939 scope.go:122] "RemoveContainer" containerID="1a3e2989d855ad99a91c6699dd9b003034fb62e249916034bdc1e5177efa9891" Oct 31 00:48:23.443797 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:48:23.443776 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3e2989d855ad99a91c6699dd9b003034fb62e249916034bdc1e5177efa9891\": container with ID starting with 1a3e2989d855ad99a91c6699dd9b003034fb62e249916034bdc1e5177efa9891 not found: ID does not exist" containerID="1a3e2989d855ad99a91c6699dd9b003034fb62e249916034bdc1e5177efa9891" Oct 31 00:48:23.443887 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.443807 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3e2989d855ad99a91c6699dd9b003034fb62e249916034bdc1e5177efa9891"} err="failed to get container status \"1a3e2989d855ad99a91c6699dd9b003034fb62e249916034bdc1e5177efa9891\": rpc error: code = NotFound desc = could not find container \"1a3e2989d855ad99a91c6699dd9b003034fb62e249916034bdc1e5177efa9891\": container with ID starting with 1a3e2989d855ad99a91c6699dd9b003034fb62e249916034bdc1e5177efa9891 not found: ID does not exist" Oct 31 00:48:23.443887 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.443824 5939 scope.go:122] "RemoveContainer" containerID="e94b110fe34d708177be646e259dff2f9952f65c75b4628a5e4bff2dd0c8a9e2" Oct 31 00:48:23.444147 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:48:23.444116 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94b110fe34d708177be646e259dff2f9952f65c75b4628a5e4bff2dd0c8a9e2\": container with ID starting with e94b110fe34d708177be646e259dff2f9952f65c75b4628a5e4bff2dd0c8a9e2 not found: ID does not exist" containerID="e94b110fe34d708177be646e259dff2f9952f65c75b4628a5e4bff2dd0c8a9e2" Oct 31 00:48:23.444147 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.444140 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94b110fe34d708177be646e259dff2f9952f65c75b4628a5e4bff2dd0c8a9e2"} err="failed to get container status \"e94b110fe34d708177be646e259dff2f9952f65c75b4628a5e4bff2dd0c8a9e2\": rpc error: code = NotFound desc = could not find container \"e94b110fe34d708177be646e259dff2f9952f65c75b4628a5e4bff2dd0c8a9e2\": container with ID starting with e94b110fe34d708177be646e259dff2f9952f65c75b4628a5e4bff2dd0c8a9e2 not found: ID does not exist" Oct 31 00:48:23.444322 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.444158 5939 scope.go:122] "RemoveContainer" containerID="01a17b5bcade8c364e93e32517887055b3fb004cd6879c08f72ed2642a0b64f4" Oct 31 00:48:23.444469 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:48:23.444448 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a17b5bcade8c364e93e32517887055b3fb004cd6879c08f72ed2642a0b64f4\": container with ID starting with 01a17b5bcade8c364e93e32517887055b3fb004cd6879c08f72ed2642a0b64f4 not found: ID does not exist" containerID="01a17b5bcade8c364e93e32517887055b3fb004cd6879c08f72ed2642a0b64f4" Oct 31 00:48:23.444558 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.444476 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a17b5bcade8c364e93e32517887055b3fb004cd6879c08f72ed2642a0b64f4"} err="failed to get container status \"01a17b5bcade8c364e93e32517887055b3fb004cd6879c08f72ed2642a0b64f4\": rpc error: code = NotFound desc = could not find container \"01a17b5bcade8c364e93e32517887055b3fb004cd6879c08f72ed2642a0b64f4\": container with ID starting with 01a17b5bcade8c364e93e32517887055b3fb004cd6879c08f72ed2642a0b64f4 not found: ID does not exist" Oct 31 00:48:23.445823 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.445804 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7"] Oct 31 00:48:23.451809 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.451790 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-xgboost-predictor-cd7fb98bb-668x7"] Oct 31 00:48:23.653193 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:23.653158 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" path="/var/lib/kubelet/pods/f85f3189-0769-4053-b252-c48e9b9afe31/volumes" Oct 31 00:48:25.788584 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.788554 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq"] Oct 31 00:48:25.789003 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.788908 5939 cpu_manager.go:414] "containerMap: removing container" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kube-rbac-proxy" Oct 31 00:48:25.789003 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.788920 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kube-rbac-proxy" Oct 31 00:48:25.789003 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.788933 5939 cpu_manager.go:414] "containerMap: removing container" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="storage-initializer" Oct 31 00:48:25.789003 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.788938 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="storage-initializer" Oct 31 00:48:25.789003 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.788949 5939 cpu_manager.go:414] "containerMap: removing container" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kserve-container" Oct 31 00:48:25.789003 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.788955 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kserve-container" Oct 31 00:48:25.789243 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.789011 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kube-rbac-proxy" Oct 31 00:48:25.789243 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.789020 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85f3189-0769-4053-b252-c48e9b9afe31" containerName="kserve-container" Oct 31 00:48:25.793827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.793808 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:25.800147 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.799953 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 00:48:25.800690 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.800648 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\"" Oct 31 00:48:25.800690 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.800682 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 00:48:25.800811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.800741 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:48:25.800978 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.800961 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 00:48:25.801062 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.801005 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-predictive-lightgbm-predictor-serving-cert\"" Oct 31 00:48:25.801123 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.801094 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 00:48:25.805419 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.805399 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq"] Oct 31 00:48:25.903599 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.903565 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/292c7b14-a324-4f93-b089-381c4875c624-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-6759789b79-zmcgq\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:25.903809 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.903610 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf49w\" (UniqueName: \"kubernetes.io/projected/292c7b14-a324-4f93-b089-381c4875c624-kube-api-access-tf49w\") pod \"isvc-predictive-lightgbm-predictor-6759789b79-zmcgq\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:25.903809 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.903635 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/292c7b14-a324-4f93-b089-381c4875c624-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-6759789b79-zmcgq\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:25.903809 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.903762 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/292c7b14-a324-4f93-b089-381c4875c624-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-6759789b79-zmcgq\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:25.903942 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:25.903827 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/292c7b14-a324-4f93-b089-381c4875c624-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-6759789b79-zmcgq\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:26.004684 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:26.004627 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/292c7b14-a324-4f93-b089-381c4875c624-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-6759789b79-zmcgq\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:26.004874 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:26.004693 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf49w\" (UniqueName: \"kubernetes.io/projected/292c7b14-a324-4f93-b089-381c4875c624-kube-api-access-tf49w\") pod \"isvc-predictive-lightgbm-predictor-6759789b79-zmcgq\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:26.004874 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:26.004719 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/292c7b14-a324-4f93-b089-381c4875c624-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-6759789b79-zmcgq\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:26.004874 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:26.004750 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/292c7b14-a324-4f93-b089-381c4875c624-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-6759789b79-zmcgq\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:26.004874 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:26.004783 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/292c7b14-a324-4f93-b089-381c4875c624-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-6759789b79-zmcgq\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:26.005223 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:26.005200 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/292c7b14-a324-4f93-b089-381c4875c624-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-6759789b79-zmcgq\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:26.005411 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:26.005394 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/292c7b14-a324-4f93-b089-381c4875c624-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-6759789b79-zmcgq\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:26.005446 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:26.005414 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/292c7b14-a324-4f93-b089-381c4875c624-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-6759789b79-zmcgq\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:26.007220 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:26.007204 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/292c7b14-a324-4f93-b089-381c4875c624-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-6759789b79-zmcgq\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:26.019227 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:26.019201 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf49w\" (UniqueName: \"kubernetes.io/projected/292c7b14-a324-4f93-b089-381c4875c624-kube-api-access-tf49w\") pod \"isvc-predictive-lightgbm-predictor-6759789b79-zmcgq\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:26.111259 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:26.111226 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:26.391447 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:26.391361 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq"] Oct 31 00:48:26.394984 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:48:26.394955 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292c7b14_a324_4f93_b089_381c4875c624.slice/crio-257ed0bcd347c07ca3329155c8a901367dced82c87498d9fd191e66ba95068b0 WatchSource:0}: Error finding container 257ed0bcd347c07ca3329155c8a901367dced82c87498d9fd191e66ba95068b0: Status 404 returned error can't find the container with id 257ed0bcd347c07ca3329155c8a901367dced82c87498d9fd191e66ba95068b0 Oct 31 00:48:26.434105 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:26.434083 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" event={"ID":"292c7b14-a324-4f93-b089-381c4875c624","Type":"ContainerStarted","Data":"257ed0bcd347c07ca3329155c8a901367dced82c87498d9fd191e66ba95068b0"} Oct 31 00:48:27.440440 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:27.440404 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" event={"ID":"292c7b14-a324-4f93-b089-381c4875c624","Type":"ContainerStarted","Data":"bb3fd15a1ddc2082ce386755aa71636182fe4a9ca88747beb017531a66ade617"} Oct 31 00:48:30.454448 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:30.454414 5939 generic.go:356] "Generic (PLEG): container finished" podID="292c7b14-a324-4f93-b089-381c4875c624" containerID="bb3fd15a1ddc2082ce386755aa71636182fe4a9ca88747beb017531a66ade617" exitCode=0 Oct 31 00:48:30.454824 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:30.454485 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" event={"ID":"292c7b14-a324-4f93-b089-381c4875c624","Type":"ContainerDied","Data":"bb3fd15a1ddc2082ce386755aa71636182fe4a9ca88747beb017531a66ade617"} Oct 31 00:48:30.512301 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:48:30.512261 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:48:31.466563 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:31.466512 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" event={"ID":"292c7b14-a324-4f93-b089-381c4875c624","Type":"ContainerStarted","Data":"b0d449c913c257edc1ab6783b4d79619cce44a4c14095aa628975494d68c6930"} Oct 31 00:48:31.466961 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:31.466782 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" event={"ID":"292c7b14-a324-4f93-b089-381c4875c624","Type":"ContainerStarted","Data":"3927317ba55748309e0acb9fa1cbda6c7530e6761946052c5d1b50c7d8779048"} Oct 31 00:48:31.467263 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:31.467240 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:31.467452 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:31.467428 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:31.468816 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:31.468790 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" podUID="292c7b14-a324-4f93-b089-381c4875c624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 31 00:48:31.496722 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:31.496651 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" podStartSLOduration=2.580948697 podStartE2EDuration="6.496640621s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.915691924s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:31.494740798 +0000 UTC m=+1123.984015978" watchObservedRunningTime="2026-10-31 00:48:31.496640621 +0000 UTC m=+1123.985915826" Oct 31 00:48:32.471059 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:32.471016 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" podUID="292c7b14-a324-4f93-b089-381c4875c624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 31 00:48:33.474987 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:33.474939 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" podUID="292c7b14-a324-4f93-b089-381c4875c624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 31 00:48:38.479951 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:38.479922 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:38.480990 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:38.480972 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:46.090348 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.090319 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq"] Oct 31 00:48:46.090790 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.090535 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" podUID="292c7b14-a324-4f93-b089-381c4875c624" containerName="kserve-container" containerID="cri-o://3927317ba55748309e0acb9fa1cbda6c7530e6761946052c5d1b50c7d8779048" gracePeriod=30 Oct 31 00:48:46.090790 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.090585 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" podUID="292c7b14-a324-4f93-b089-381c4875c624" containerName="kube-rbac-proxy" containerID="cri-o://b0d449c913c257edc1ab6783b4d79619cce44a4c14095aa628975494d68c6930" gracePeriod=30 Oct 31 00:48:46.530830 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.530792 5939 generic.go:356] "Generic (PLEG): container finished" podID="292c7b14-a324-4f93-b089-381c4875c624" containerID="b0d449c913c257edc1ab6783b4d79619cce44a4c14095aa628975494d68c6930" exitCode=0 Oct 31 00:48:46.531019 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.530852 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" event={"ID":"292c7b14-a324-4f93-b089-381c4875c624","Type":"ContainerDied","Data":"b0d449c913c257edc1ab6783b4d79619cce44a4c14095aa628975494d68c6930"} Oct 31 00:48:46.832193 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.832163 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:46.870836 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.870806 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/292c7b14-a324-4f93-b089-381c4875c624-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/292c7b14-a324-4f93-b089-381c4875c624-cabundle-cert\") pod \"292c7b14-a324-4f93-b089-381c4875c624\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " Oct 31 00:48:46.870836 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.870840 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/292c7b14-a324-4f93-b089-381c4875c624-kube-api-access-tf49w\" (UniqueName: \"kubernetes.io/projected/292c7b14-a324-4f93-b089-381c4875c624-kube-api-access-tf49w\") pod \"292c7b14-a324-4f93-b089-381c4875c624\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " Oct 31 00:48:46.871038 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.870888 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/292c7b14-a324-4f93-b089-381c4875c624-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/292c7b14-a324-4f93-b089-381c4875c624-kserve-provision-location\") pod \"292c7b14-a324-4f93-b089-381c4875c624\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " Oct 31 00:48:46.871038 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.870963 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/292c7b14-a324-4f93-b089-381c4875c624-proxy-tls\" (UniqueName: \"kubernetes.io/secret/292c7b14-a324-4f93-b089-381c4875c624-proxy-tls\") pod \"292c7b14-a324-4f93-b089-381c4875c624\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " Oct 31 00:48:46.871038 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.871016 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/292c7b14-a324-4f93-b089-381c4875c624-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/292c7b14-a324-4f93-b089-381c4875c624-kube-rbac-proxy-sar-config\") pod \"292c7b14-a324-4f93-b089-381c4875c624\" (UID: \"292c7b14-a324-4f93-b089-381c4875c624\") " Oct 31 00:48:46.871202 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.871184 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292c7b14-a324-4f93-b089-381c4875c624-cabundle-cert" pod "292c7b14-a324-4f93-b089-381c4875c624" (UID: "292c7b14-a324-4f93-b089-381c4875c624"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:46.871269 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.871241 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292c7b14-a324-4f93-b089-381c4875c624-kserve-provision-location" pod "292c7b14-a324-4f93-b089-381c4875c624" (UID: "292c7b14-a324-4f93-b089-381c4875c624"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:46.871322 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.871252 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/292c7b14-a324-4f93-b089-381c4875c624-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:46.871450 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.871425 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292c7b14-a324-4f93-b089-381c4875c624-kube-rbac-proxy-sar-config" pod "292c7b14-a324-4f93-b089-381c4875c624" (UID: "292c7b14-a324-4f93-b089-381c4875c624"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:46.872975 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.872944 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292c7b14-a324-4f93-b089-381c4875c624-proxy-tls" pod "292c7b14-a324-4f93-b089-381c4875c624" (UID: "292c7b14-a324-4f93-b089-381c4875c624"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:46.873089 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.873002 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/292c7b14-a324-4f93-b089-381c4875c624-kube-api-access-tf49w" pod "292c7b14-a324-4f93-b089-381c4875c624" (UID: "292c7b14-a324-4f93-b089-381c4875c624"). InnerVolumeSpecName "kube-api-access-tf49w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:46.972546 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.972515 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf49w\" (UniqueName: \"kubernetes.io/projected/292c7b14-a324-4f93-b089-381c4875c624-kube-api-access-tf49w\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:46.972546 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.972542 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/292c7b14-a324-4f93-b089-381c4875c624-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:46.972742 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.972572 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/292c7b14-a324-4f93-b089-381c4875c624-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:46.972742 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:46.972582 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/292c7b14-a324-4f93-b089-381c4875c624-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:47.537318 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:47.537279 5939 generic.go:356] "Generic (PLEG): container finished" podID="292c7b14-a324-4f93-b089-381c4875c624" containerID="3927317ba55748309e0acb9fa1cbda6c7530e6761946052c5d1b50c7d8779048" exitCode=0 Oct 31 00:48:47.537849 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:47.537348 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" Oct 31 00:48:47.537849 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:47.537366 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" event={"ID":"292c7b14-a324-4f93-b089-381c4875c624","Type":"ContainerDied","Data":"3927317ba55748309e0acb9fa1cbda6c7530e6761946052c5d1b50c7d8779048"} Oct 31 00:48:47.537849 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:47.537433 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq" event={"ID":"292c7b14-a324-4f93-b089-381c4875c624","Type":"ContainerDied","Data":"257ed0bcd347c07ca3329155c8a901367dced82c87498d9fd191e66ba95068b0"} Oct 31 00:48:47.537849 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:47.537459 5939 scope.go:122] "RemoveContainer" containerID="b0d449c913c257edc1ab6783b4d79619cce44a4c14095aa628975494d68c6930" Oct 31 00:48:47.546583 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:47.546563 5939 scope.go:122] "RemoveContainer" containerID="3927317ba55748309e0acb9fa1cbda6c7530e6761946052c5d1b50c7d8779048" Oct 31 00:48:47.553761 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:47.553743 5939 scope.go:122] "RemoveContainer" containerID="bb3fd15a1ddc2082ce386755aa71636182fe4a9ca88747beb017531a66ade617" Oct 31 00:48:47.560694 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:47.560677 5939 scope.go:122] "RemoveContainer" containerID="b0d449c913c257edc1ab6783b4d79619cce44a4c14095aa628975494d68c6930" Oct 31 00:48:47.560922 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:48:47.560904 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d449c913c257edc1ab6783b4d79619cce44a4c14095aa628975494d68c6930\": container with ID starting with b0d449c913c257edc1ab6783b4d79619cce44a4c14095aa628975494d68c6930 not found: ID does not exist" containerID="b0d449c913c257edc1ab6783b4d79619cce44a4c14095aa628975494d68c6930" Oct 31 00:48:47.560969 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:47.560931 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d449c913c257edc1ab6783b4d79619cce44a4c14095aa628975494d68c6930"} err="failed to get container status \"b0d449c913c257edc1ab6783b4d79619cce44a4c14095aa628975494d68c6930\": rpc error: code = NotFound desc = could not find container \"b0d449c913c257edc1ab6783b4d79619cce44a4c14095aa628975494d68c6930\": container with ID starting with b0d449c913c257edc1ab6783b4d79619cce44a4c14095aa628975494d68c6930 not found: ID does not exist" Oct 31 00:48:47.560969 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:47.560947 5939 scope.go:122] "RemoveContainer" containerID="3927317ba55748309e0acb9fa1cbda6c7530e6761946052c5d1b50c7d8779048" Oct 31 00:48:47.561138 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:48:47.561122 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3927317ba55748309e0acb9fa1cbda6c7530e6761946052c5d1b50c7d8779048\": container with ID starting with 3927317ba55748309e0acb9fa1cbda6c7530e6761946052c5d1b50c7d8779048 not found: ID does not exist" containerID="3927317ba55748309e0acb9fa1cbda6c7530e6761946052c5d1b50c7d8779048" Oct 31 00:48:47.561178 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:47.561147 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3927317ba55748309e0acb9fa1cbda6c7530e6761946052c5d1b50c7d8779048"} err="failed to get container status \"3927317ba55748309e0acb9fa1cbda6c7530e6761946052c5d1b50c7d8779048\": rpc error: code = NotFound desc = could not find container \"3927317ba55748309e0acb9fa1cbda6c7530e6761946052c5d1b50c7d8779048\": container with ID starting with 3927317ba55748309e0acb9fa1cbda6c7530e6761946052c5d1b50c7d8779048 not found: ID does not exist" Oct 31 00:48:47.561178 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:47.561161 5939 scope.go:122] "RemoveContainer" containerID="bb3fd15a1ddc2082ce386755aa71636182fe4a9ca88747beb017531a66ade617" Oct 31 00:48:47.561358 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:48:47.561342 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3fd15a1ddc2082ce386755aa71636182fe4a9ca88747beb017531a66ade617\": container with ID starting with bb3fd15a1ddc2082ce386755aa71636182fe4a9ca88747beb017531a66ade617 not found: ID does not exist" containerID="bb3fd15a1ddc2082ce386755aa71636182fe4a9ca88747beb017531a66ade617" Oct 31 00:48:47.561395 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:47.561364 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3fd15a1ddc2082ce386755aa71636182fe4a9ca88747beb017531a66ade617"} err="failed to get container status \"bb3fd15a1ddc2082ce386755aa71636182fe4a9ca88747beb017531a66ade617\": rpc error: code = NotFound desc = could not find container \"bb3fd15a1ddc2082ce386755aa71636182fe4a9ca88747beb017531a66ade617\": container with ID starting with bb3fd15a1ddc2082ce386755aa71636182fe4a9ca88747beb017531a66ade617 not found: ID does not exist" Oct 31 00:48:47.563546 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:47.563527 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq"] Oct 31 00:48:47.571605 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:47.571585 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-lightgbm-predictor-6759789b79-zmcgq"] Oct 31 00:48:47.652097 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:47.652056 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="292c7b14-a324-4f93-b089-381c4875c624" path="/var/lib/kubelet/pods/292c7b14-a324-4f93-b089-381c4875c624/volumes" Oct 31 00:48:49.480592 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.480558 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m"] Oct 31 00:48:49.481007 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.480900 5939 cpu_manager.go:414] "containerMap: removing container" podUID="292c7b14-a324-4f93-b089-381c4875c624" containerName="kserve-container" Oct 31 00:48:49.481007 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.480911 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="292c7b14-a324-4f93-b089-381c4875c624" containerName="kserve-container" Oct 31 00:48:49.481007 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.480924 5939 cpu_manager.go:414] "containerMap: removing container" podUID="292c7b14-a324-4f93-b089-381c4875c624" containerName="storage-initializer" Oct 31 00:48:49.481007 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.480930 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="292c7b14-a324-4f93-b089-381c4875c624" containerName="storage-initializer" Oct 31 00:48:49.481007 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.480944 5939 cpu_manager.go:414] "containerMap: removing container" podUID="292c7b14-a324-4f93-b089-381c4875c624" containerName="kube-rbac-proxy" Oct 31 00:48:49.481007 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.480949 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="292c7b14-a324-4f93-b089-381c4875c624" containerName="kube-rbac-proxy" Oct 31 00:48:49.481007 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.480998 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="292c7b14-a324-4f93-b089-381c4875c624" containerName="kserve-container" Oct 31 00:48:49.481007 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.481007 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="292c7b14-a324-4f93-b089-381c4875c624" containerName="kube-rbac-proxy" Oct 31 00:48:49.486118 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.486099 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:49.494035 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.494012 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\"" Oct 31 00:48:49.494035 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.494016 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 00:48:49.494413 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.494390 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 00:48:49.494413 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.494404 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:48:49.494570 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.494511 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\"" Oct 31 00:48:49.494570 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.494546 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 00:48:49.494651 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.494572 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 00:48:49.497596 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.497569 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m"] Oct 31 00:48:49.594517 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.594485 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:49.594733 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.594526 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4kn9\" (UniqueName: \"kubernetes.io/projected/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kube-api-access-d4kn9\") pod \"isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:49.594733 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.594553 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:49.594733 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.594660 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:49.594872 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.594756 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:49.695140 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.695108 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:49.695310 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.695149 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:49.695310 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.695191 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:49.695310 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.695223 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4kn9\" (UniqueName: \"kubernetes.io/projected/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kube-api-access-d4kn9\") pod \"isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:49.695310 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.695246 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:49.695719 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.695695 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:49.695921 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.695902 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:49.695970 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.695959 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:49.697555 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.697532 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:49.709718 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.709689 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4kn9\" (UniqueName: \"kubernetes.io/projected/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kube-api-access-d4kn9\") pod \"isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:49.803518 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:49.803440 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:50.086092 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:50.085959 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m"] Oct 31 00:48:50.088754 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:48:50.088719 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f59a3f1_f9e3_4bc6_a2a1_5722b2db7383.slice/crio-190846ba7d31f908ca6fa41549e00320a8d0db939c75d1df2221799658b4e43a WatchSource:0}: Error finding container 190846ba7d31f908ca6fa41549e00320a8d0db939c75d1df2221799658b4e43a: Status 404 returned error can't find the container with id 190846ba7d31f908ca6fa41549e00320a8d0db939c75d1df2221799658b4e43a Oct 31 00:48:50.551529 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:50.551492 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" event={"ID":"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383","Type":"ContainerStarted","Data":"e0afeb303edc9fa181992d5c1c39c1c477298cde5c12cdd64bf58a1a54fc9573"} Oct 31 00:48:50.551931 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:50.551549 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" event={"ID":"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383","Type":"ContainerStarted","Data":"190846ba7d31f908ca6fa41549e00320a8d0db939c75d1df2221799658b4e43a"} Oct 31 00:48:54.566980 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:54.566944 5939 generic.go:356] "Generic (PLEG): container finished" podID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerID="e0afeb303edc9fa181992d5c1c39c1c477298cde5c12cdd64bf58a1a54fc9573" exitCode=0 Oct 31 00:48:54.567345 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:54.567016 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" event={"ID":"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383","Type":"ContainerDied","Data":"e0afeb303edc9fa181992d5c1c39c1c477298cde5c12cdd64bf58a1a54fc9573"} Oct 31 00:48:55.572895 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:55.572860 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" event={"ID":"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383","Type":"ContainerStarted","Data":"e082ac6bfd50c197d1b9c105a51eb115e5d266a79cbc52f7f7476754efcf0af5"} Oct 31 00:48:55.573488 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:55.572931 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" event={"ID":"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383","Type":"ContainerStarted","Data":"4c46a511362429ad8b21cdc8529e1ad5273c47988f88e89bb4f0d9dcc700044a"} Oct 31 00:48:55.573488 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:55.573148 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:48:55.604995 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:55.604950 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" podStartSLOduration=2.850971869 podStartE2EDuration="6.604938809s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.75396694s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:55.603263939 +0000 UTC m=+1148.092539118" watchObservedRunningTime="2026-10-31 00:48:55.604938809 +0000 UTC m=+1148.094213981" Oct 31 00:48:56.576702 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:48:56.576659 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:49:00.508237 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:49:00.508199 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:49:02.585159 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:49:02.585129 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:49:30.509384 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:49:30.509307 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:49:32.586258 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:49:32.586216 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" podUID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.35:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.35:8080: connect: connection refused" Oct 31 00:49:42.586179 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:49:42.586140 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" podUID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.35:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.35:8080: connect: connection refused" Oct 31 00:49:47.785322 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:49:47.785292 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:49:47.786120 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:49:47.786100 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:49:52.585993 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:49:52.585953 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" podUID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.35:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.35:8080: connect: connection refused" Oct 31 00:50:00.509414 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:50:00.509372 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:50:03.021599 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:03.021568 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:50:09.950117 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:09.950086 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m"] Oct 31 00:50:09.950537 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:09.950311 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" podUID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerName="kserve-container" containerID="cri-o://4c46a511362429ad8b21cdc8529e1ad5273c47988f88e89bb4f0d9dcc700044a" gracePeriod=30 Oct 31 00:50:09.950537 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:09.950370 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" podUID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerName="kube-rbac-proxy" containerID="cri-o://e082ac6bfd50c197d1b9c105a51eb115e5d266a79cbc52f7f7476754efcf0af5" gracePeriod=30 Oct 31 00:50:10.862086 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:10.862054 5939 generic.go:356] "Generic (PLEG): container finished" podID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerID="e082ac6bfd50c197d1b9c105a51eb115e5d266a79cbc52f7f7476754efcf0af5" exitCode=0 Oct 31 00:50:10.862255 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:10.862101 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" event={"ID":"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383","Type":"ContainerDied","Data":"e082ac6bfd50c197d1b9c105a51eb115e5d266a79cbc52f7f7476754efcf0af5"} Oct 31 00:50:12.580764 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:12.580719 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" podUID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused" Oct 31 00:50:12.586121 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:12.586092 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" podUID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.35:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.35:8080: connect: connection refused" Oct 31 00:50:14.684471 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.684446 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:50:14.806954 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.806870 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kube-api-access-d4kn9\" (UniqueName: \"kubernetes.io/projected/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kube-api-access-d4kn9\") pod \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " Oct 31 00:50:14.806954 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.806916 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kube-rbac-proxy-sar-config\") pod \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " Oct 31 00:50:14.807156 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.806964 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-cabundle-cert\") pod \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " Oct 31 00:50:14.807156 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.807072 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kserve-provision-location\") pod \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " Oct 31 00:50:14.807156 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.807113 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-proxy-tls\") pod \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\" (UID: \"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383\") " Oct 31 00:50:14.807329 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.807303 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kube-rbac-proxy-sar-config" pod "3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" (UID: "3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:14.807329 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.807317 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-cabundle-cert" pod "3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" (UID: "3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:14.807444 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.807365 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kserve-provision-location" pod "3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" (UID: "3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:14.807504 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.807447 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:14.807504 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.807460 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:14.807504 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.807471 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:14.809175 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.809155 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kube-api-access-d4kn9" pod "3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" (UID: "3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383"). InnerVolumeSpecName "kube-api-access-d4kn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:14.809288 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.809267 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-proxy-tls" pod "3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" (UID: "3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:14.879588 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.879549 5939 generic.go:356] "Generic (PLEG): container finished" podID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerID="4c46a511362429ad8b21cdc8529e1ad5273c47988f88e89bb4f0d9dcc700044a" exitCode=0 Oct 31 00:50:14.879753 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.879600 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" Oct 31 00:50:14.879753 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.879630 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" event={"ID":"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383","Type":"ContainerDied","Data":"4c46a511362429ad8b21cdc8529e1ad5273c47988f88e89bb4f0d9dcc700044a"} Oct 31 00:50:14.879753 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.879727 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m" event={"ID":"3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383","Type":"ContainerDied","Data":"190846ba7d31f908ca6fa41549e00320a8d0db939c75d1df2221799658b4e43a"} Oct 31 00:50:14.879753 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.879754 5939 scope.go:122] "RemoveContainer" containerID="e082ac6bfd50c197d1b9c105a51eb115e5d266a79cbc52f7f7476754efcf0af5" Oct 31 00:50:14.888250 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.888235 5939 scope.go:122] "RemoveContainer" containerID="4c46a511362429ad8b21cdc8529e1ad5273c47988f88e89bb4f0d9dcc700044a" Oct 31 00:50:14.895649 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.895632 5939 scope.go:122] "RemoveContainer" containerID="e0afeb303edc9fa181992d5c1c39c1c477298cde5c12cdd64bf58a1a54fc9573" Oct 31 00:50:14.902993 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.902974 5939 scope.go:122] "RemoveContainer" containerID="e082ac6bfd50c197d1b9c105a51eb115e5d266a79cbc52f7f7476754efcf0af5" Oct 31 00:50:14.903263 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:50:14.903238 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e082ac6bfd50c197d1b9c105a51eb115e5d266a79cbc52f7f7476754efcf0af5\": container with ID starting with e082ac6bfd50c197d1b9c105a51eb115e5d266a79cbc52f7f7476754efcf0af5 not found: ID does not exist" containerID="e082ac6bfd50c197d1b9c105a51eb115e5d266a79cbc52f7f7476754efcf0af5" Oct 31 00:50:14.903378 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.903269 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e082ac6bfd50c197d1b9c105a51eb115e5d266a79cbc52f7f7476754efcf0af5"} err="failed to get container status \"e082ac6bfd50c197d1b9c105a51eb115e5d266a79cbc52f7f7476754efcf0af5\": rpc error: code = NotFound desc = could not find container \"e082ac6bfd50c197d1b9c105a51eb115e5d266a79cbc52f7f7476754efcf0af5\": container with ID starting with e082ac6bfd50c197d1b9c105a51eb115e5d266a79cbc52f7f7476754efcf0af5 not found: ID does not exist" Oct 31 00:50:14.903378 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.903294 5939 scope.go:122] "RemoveContainer" containerID="4c46a511362429ad8b21cdc8529e1ad5273c47988f88e89bb4f0d9dcc700044a" Oct 31 00:50:14.903616 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:50:14.903590 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c46a511362429ad8b21cdc8529e1ad5273c47988f88e89bb4f0d9dcc700044a\": container with ID starting with 4c46a511362429ad8b21cdc8529e1ad5273c47988f88e89bb4f0d9dcc700044a not found: ID does not exist" containerID="4c46a511362429ad8b21cdc8529e1ad5273c47988f88e89bb4f0d9dcc700044a" Oct 31 00:50:14.903718 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.903625 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c46a511362429ad8b21cdc8529e1ad5273c47988f88e89bb4f0d9dcc700044a"} err="failed to get container status \"4c46a511362429ad8b21cdc8529e1ad5273c47988f88e89bb4f0d9dcc700044a\": rpc error: code = NotFound desc = could not find container \"4c46a511362429ad8b21cdc8529e1ad5273c47988f88e89bb4f0d9dcc700044a\": container with ID starting with 4c46a511362429ad8b21cdc8529e1ad5273c47988f88e89bb4f0d9dcc700044a not found: ID does not exist" Oct 31 00:50:14.903718 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.903643 5939 scope.go:122] "RemoveContainer" containerID="e0afeb303edc9fa181992d5c1c39c1c477298cde5c12cdd64bf58a1a54fc9573" Oct 31 00:50:14.904029 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:50:14.903955 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0afeb303edc9fa181992d5c1c39c1c477298cde5c12cdd64bf58a1a54fc9573\": container with ID starting with e0afeb303edc9fa181992d5c1c39c1c477298cde5c12cdd64bf58a1a54fc9573 not found: ID does not exist" containerID="e0afeb303edc9fa181992d5c1c39c1c477298cde5c12cdd64bf58a1a54fc9573" Oct 31 00:50:14.904029 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.903985 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0afeb303edc9fa181992d5c1c39c1c477298cde5c12cdd64bf58a1a54fc9573"} err="failed to get container status \"e0afeb303edc9fa181992d5c1c39c1c477298cde5c12cdd64bf58a1a54fc9573\": rpc error: code = NotFound desc = could not find container \"e0afeb303edc9fa181992d5c1c39c1c477298cde5c12cdd64bf58a1a54fc9573\": container with ID starting with e0afeb303edc9fa181992d5c1c39c1c477298cde5c12cdd64bf58a1a54fc9573 not found: ID does not exist" Oct 31 00:50:14.905281 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.905264 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m"] Oct 31 00:50:14.908483 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.908466 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4kn9\" (UniqueName: \"kubernetes.io/projected/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-kube-api-access-d4kn9\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:14.908547 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.908485 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:14.910897 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:14.910878 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-sklearn-v2-predictor-75fc55d9c9-d8x8m"] Oct 31 00:50:15.313009 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.312975 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm"] Oct 31 00:50:15.313322 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.313309 5939 cpu_manager.go:414] "containerMap: removing container" podUID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerName="kube-rbac-proxy" Oct 31 00:50:15.313363 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.313324 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerName="kube-rbac-proxy" Oct 31 00:50:15.313363 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.313337 5939 cpu_manager.go:414] "containerMap: removing container" podUID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerName="kserve-container" Oct 31 00:50:15.313363 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.313342 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerName="kserve-container" Oct 31 00:50:15.313363 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.313353 5939 cpu_manager.go:414] "containerMap: removing container" podUID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerName="storage-initializer" Oct 31 00:50:15.313363 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.313358 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerName="storage-initializer" Oct 31 00:50:15.313511 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.313406 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerName="kube-rbac-proxy" Oct 31 00:50:15.313511 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.313416 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" containerName="kserve-container" Oct 31 00:50:15.318085 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.318065 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:15.326111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.326087 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 00:50:15.326111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.326098 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:50:15.326268 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.326144 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\"" Oct 31 00:50:15.326268 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.326258 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 00:50:15.326268 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.326261 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\"" Oct 31 00:50:15.326577 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.326566 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 00:50:15.326611 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.326583 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 00:50:15.329769 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.329750 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm"] Oct 31 00:50:15.413536 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.413500 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f5d541a-716e-4e31-a657-d6956772fa34-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:15.413536 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.413539 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28njl\" (UniqueName: \"kubernetes.io/projected/9f5d541a-716e-4e31-a657-d6956772fa34-kube-api-access-28njl\") pod \"isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:15.413832 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.413564 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f5d541a-716e-4e31-a657-d6956772fa34-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:15.413832 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.413645 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f5d541a-716e-4e31-a657-d6956772fa34-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:15.413832 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.413781 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f5d541a-716e-4e31-a657-d6956772fa34-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:15.515001 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.514962 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f5d541a-716e-4e31-a657-d6956772fa34-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:15.515166 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.515007 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28njl\" (UniqueName: \"kubernetes.io/projected/9f5d541a-716e-4e31-a657-d6956772fa34-kube-api-access-28njl\") pod \"isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:15.515166 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.515028 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f5d541a-716e-4e31-a657-d6956772fa34-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:15.515166 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.515138 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f5d541a-716e-4e31-a657-d6956772fa34-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:15.515337 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.515234 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f5d541a-716e-4e31-a657-d6956772fa34-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:15.515446 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.515426 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f5d541a-716e-4e31-a657-d6956772fa34-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:15.515806 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.515785 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f5d541a-716e-4e31-a657-d6956772fa34-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:15.515909 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.515792 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f5d541a-716e-4e31-a657-d6956772fa34-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:15.517468 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.517444 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f5d541a-716e-4e31-a657-d6956772fa34-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:15.529649 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.529632 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28njl\" (UniqueName: \"kubernetes.io/projected/9f5d541a-716e-4e31-a657-d6956772fa34-kube-api-access-28njl\") pod \"isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:15.635043 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.635005 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:15.652742 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.652713 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383" path="/var/lib/kubelet/pods/3f59a3f1-f9e3-4bc6-a2a1-5722b2db7383/volumes" Oct 31 00:50:15.915023 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:15.914947 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm"] Oct 31 00:50:15.918091 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:50:15.918061 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5d541a_716e_4e31_a657_d6956772fa34.slice/crio-d7cf14e9a820f21468050494d4149326e99e9c1431f59d2e2babd3f950c3f815 WatchSource:0}: Error finding container d7cf14e9a820f21468050494d4149326e99e9c1431f59d2e2babd3f950c3f815: Status 404 returned error can't find the container with id d7cf14e9a820f21468050494d4149326e99e9c1431f59d2e2babd3f950c3f815 Oct 31 00:50:16.890110 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:16.890073 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" event={"ID":"9f5d541a-716e-4e31-a657-d6956772fa34","Type":"ContainerStarted","Data":"4bc2962e94d375a14b04376a4b345a8eeb41b8c494752e1bca55826cf5b5a7c8"} Oct 31 00:50:16.890328 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:16.890129 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" event={"ID":"9f5d541a-716e-4e31-a657-d6956772fa34","Type":"ContainerStarted","Data":"d7cf14e9a820f21468050494d4149326e99e9c1431f59d2e2babd3f950c3f815"} Oct 31 00:50:19.905427 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:19.905339 5939 generic.go:356] "Generic (PLEG): container finished" podID="9f5d541a-716e-4e31-a657-d6956772fa34" containerID="4bc2962e94d375a14b04376a4b345a8eeb41b8c494752e1bca55826cf5b5a7c8" exitCode=0 Oct 31 00:50:19.905427 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:19.905411 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" event={"ID":"9f5d541a-716e-4e31-a657-d6956772fa34","Type":"ContainerDied","Data":"4bc2962e94d375a14b04376a4b345a8eeb41b8c494752e1bca55826cf5b5a7c8"} Oct 31 00:50:20.912235 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:20.912200 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" event={"ID":"9f5d541a-716e-4e31-a657-d6956772fa34","Type":"ContainerStarted","Data":"d225d3e5f37278016aee04ad1b61f2a02d5774b61bdac3b3afcaa1cc8971d6af"} Oct 31 00:50:20.912646 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:20.912259 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" event={"ID":"9f5d541a-716e-4e31-a657-d6956772fa34","Type":"ContainerStarted","Data":"ed897f4f5afc1b3205a3fb1522530b5759fba90002192c7e836156501257e209"} Oct 31 00:50:20.912646 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:20.912453 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:20.941597 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:20.941539 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" podStartSLOduration=2.265891368 podStartE2EDuration="5.941523166s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.675631798s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:20.940288935 +0000 UTC m=+1233.429564106" watchObservedRunningTime="2026-10-31 00:50:20.941523166 +0000 UTC m=+1233.430798324" Oct 31 00:50:21.916470 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:21.916443 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:27.925794 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:27.925766 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:50:30.509894 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:50:30.509853 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:50:57.926772 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:50:57.926729 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" podUID="9f5d541a-716e-4e31-a657-d6956772fa34" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused" Oct 31 00:51:00.509350 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:51:00.509304 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:51:07.926294 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:07.926255 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" podUID="9f5d541a-716e-4e31-a657-d6956772fa34" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused" Oct 31 00:51:17.926270 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:17.926230 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" podUID="9f5d541a-716e-4e31-a657-d6956772fa34" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused" Oct 31 00:51:28.514213 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:28.514185 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:51:30.509076 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:51:30.509035 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:51:35.673218 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:35.673188 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm"] Oct 31 00:51:35.673624 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:35.673411 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" podUID="9f5d541a-716e-4e31-a657-d6956772fa34" containerName="kserve-container" containerID="cri-o://ed897f4f5afc1b3205a3fb1522530b5759fba90002192c7e836156501257e209" gracePeriod=30 Oct 31 00:51:35.673624 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:35.673508 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" podUID="9f5d541a-716e-4e31-a657-d6956772fa34" containerName="kube-rbac-proxy" containerID="cri-o://d225d3e5f37278016aee04ad1b61f2a02d5774b61bdac3b3afcaa1cc8971d6af" gracePeriod=30 Oct 31 00:51:36.218857 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:36.218823 5939 generic.go:356] "Generic (PLEG): container finished" podID="9f5d541a-716e-4e31-a657-d6956772fa34" containerID="d225d3e5f37278016aee04ad1b61f2a02d5774b61bdac3b3afcaa1cc8971d6af" exitCode=0 Oct 31 00:51:36.218857 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:36.218860 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" event={"ID":"9f5d541a-716e-4e31-a657-d6956772fa34","Type":"ContainerDied","Data":"d225d3e5f37278016aee04ad1b61f2a02d5774b61bdac3b3afcaa1cc8971d6af"} Oct 31 00:51:37.922099 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:37.922057 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" podUID="9f5d541a-716e-4e31-a657-d6956772fa34" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused" Oct 31 00:51:37.926486 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:37.926452 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" podUID="9f5d541a-716e-4e31-a657-d6956772fa34" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused" Oct 31 00:51:40.239219 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.239187 5939 generic.go:356] "Generic (PLEG): container finished" podID="9f5d541a-716e-4e31-a657-d6956772fa34" containerID="ed897f4f5afc1b3205a3fb1522530b5759fba90002192c7e836156501257e209" exitCode=0 Oct 31 00:51:40.239557 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.239261 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" event={"ID":"9f5d541a-716e-4e31-a657-d6956772fa34","Type":"ContainerDied","Data":"ed897f4f5afc1b3205a3fb1522530b5759fba90002192c7e836156501257e209"} Oct 31 00:51:40.310907 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.310882 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:51:40.346580 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.346551 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f5d541a-716e-4e31-a657-d6956772fa34-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f5d541a-716e-4e31-a657-d6956772fa34-cabundle-cert\") pod \"9f5d541a-716e-4e31-a657-d6956772fa34\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " Oct 31 00:51:40.346749 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.346598 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f5d541a-716e-4e31-a657-d6956772fa34-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f5d541a-716e-4e31-a657-d6956772fa34-kserve-provision-location\") pod \"9f5d541a-716e-4e31-a657-d6956772fa34\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " Oct 31 00:51:40.346749 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.346645 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f5d541a-716e-4e31-a657-d6956772fa34-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f5d541a-716e-4e31-a657-d6956772fa34-proxy-tls\") pod \"9f5d541a-716e-4e31-a657-d6956772fa34\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " Oct 31 00:51:40.346749 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.346698 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f5d541a-716e-4e31-a657-d6956772fa34-kube-api-access-28njl\" (UniqueName: \"kubernetes.io/projected/9f5d541a-716e-4e31-a657-d6956772fa34-kube-api-access-28njl\") pod \"9f5d541a-716e-4e31-a657-d6956772fa34\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " Oct 31 00:51:40.346749 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.346743 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f5d541a-716e-4e31-a657-d6956772fa34-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f5d541a-716e-4e31-a657-d6956772fa34-kube-rbac-proxy-sar-config\") pod \"9f5d541a-716e-4e31-a657-d6956772fa34\" (UID: \"9f5d541a-716e-4e31-a657-d6956772fa34\") " Oct 31 00:51:40.347071 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.346965 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5d541a-716e-4e31-a657-d6956772fa34-kserve-provision-location" pod "9f5d541a-716e-4e31-a657-d6956772fa34" (UID: "9f5d541a-716e-4e31-a657-d6956772fa34"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:40.347071 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.347003 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5d541a-716e-4e31-a657-d6956772fa34-cabundle-cert" pod "9f5d541a-716e-4e31-a657-d6956772fa34" (UID: "9f5d541a-716e-4e31-a657-d6956772fa34"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:40.347242 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.347166 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5d541a-716e-4e31-a657-d6956772fa34-kube-rbac-proxy-sar-config" pod "9f5d541a-716e-4e31-a657-d6956772fa34" (UID: "9f5d541a-716e-4e31-a657-d6956772fa34"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:40.348892 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.348866 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5d541a-716e-4e31-a657-d6956772fa34-kube-api-access-28njl" pod "9f5d541a-716e-4e31-a657-d6956772fa34" (UID: "9f5d541a-716e-4e31-a657-d6956772fa34"). InnerVolumeSpecName "kube-api-access-28njl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:40.348991 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.348866 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f5d541a-716e-4e31-a657-d6956772fa34-proxy-tls" pod "9f5d541a-716e-4e31-a657-d6956772fa34" (UID: "9f5d541a-716e-4e31-a657-d6956772fa34"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:40.447541 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.447466 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f5d541a-716e-4e31-a657-d6956772fa34-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.447541 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.447492 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f5d541a-716e-4e31-a657-d6956772fa34-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.447541 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.447503 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f5d541a-716e-4e31-a657-d6956772fa34-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.447541 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.447512 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28njl\" (UniqueName: \"kubernetes.io/projected/9f5d541a-716e-4e31-a657-d6956772fa34-kube-api-access-28njl\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.447541 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:40.447520 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f5d541a-716e-4e31-a657-d6956772fa34-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:41.245136 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:41.245109 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" Oct 31 00:51:41.245537 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:41.245103 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm" event={"ID":"9f5d541a-716e-4e31-a657-d6956772fa34","Type":"ContainerDied","Data":"d7cf14e9a820f21468050494d4149326e99e9c1431f59d2e2babd3f950c3f815"} Oct 31 00:51:41.245537 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:41.245255 5939 scope.go:122] "RemoveContainer" containerID="d225d3e5f37278016aee04ad1b61f2a02d5774b61bdac3b3afcaa1cc8971d6af" Oct 31 00:51:41.254458 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:41.254436 5939 scope.go:122] "RemoveContainer" containerID="ed897f4f5afc1b3205a3fb1522530b5759fba90002192c7e836156501257e209" Oct 31 00:51:41.261889 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:41.261873 5939 scope.go:122] "RemoveContainer" containerID="4bc2962e94d375a14b04376a4b345a8eeb41b8c494752e1bca55826cf5b5a7c8" Oct 31 00:51:41.271059 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:41.271037 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm"] Oct 31 00:51:41.277125 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:41.277104 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-xgboost-v2-predictor-686dc9956-rdtbm"] Oct 31 00:51:41.651904 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:41.651868 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f5d541a-716e-4e31-a657-d6956772fa34" path="/var/lib/kubelet/pods/9f5d541a-716e-4e31-a657-d6956772fa34/volumes" Oct 31 00:51:42.663174 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.663149 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl"] Oct 31 00:51:42.663554 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.663454 5939 cpu_manager.go:414] "containerMap: removing container" podUID="9f5d541a-716e-4e31-a657-d6956772fa34" containerName="kube-rbac-proxy" Oct 31 00:51:42.663554 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.663464 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5d541a-716e-4e31-a657-d6956772fa34" containerName="kube-rbac-proxy" Oct 31 00:51:42.663554 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.663473 5939 cpu_manager.go:414] "containerMap: removing container" podUID="9f5d541a-716e-4e31-a657-d6956772fa34" containerName="kserve-container" Oct 31 00:51:42.663554 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.663479 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5d541a-716e-4e31-a657-d6956772fa34" containerName="kserve-container" Oct 31 00:51:42.663554 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.663500 5939 cpu_manager.go:414] "containerMap: removing container" podUID="9f5d541a-716e-4e31-a657-d6956772fa34" containerName="storage-initializer" Oct 31 00:51:42.663554 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.663505 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5d541a-716e-4e31-a657-d6956772fa34" containerName="storage-initializer" Oct 31 00:51:42.663554 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.663554 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5d541a-716e-4e31-a657-d6956772fa34" containerName="kube-rbac-proxy" Oct 31 00:51:42.663806 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.663563 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5d541a-716e-4e31-a657-d6956772fa34" containerName="kserve-container" Oct 31 00:51:42.668081 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.668062 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl" Oct 31 00:51:42.676781 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.676756 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl"] Oct 31 00:51:42.677251 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.677229 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\"" Oct 31 00:51:42.677416 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.677401 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\"" Oct 31 00:51:42.677508 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.677493 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\"" Oct 31 00:51:42.766802 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.766765 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cn4s\" (UniqueName: \"kubernetes.io/projected/e8f84625-11a2-4de1-8d23-45555705aedd-kube-api-access-9cn4s\") pod \"proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl\" (UID: \"e8f84625-11a2-4de1-8d23-45555705aedd\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl" Oct 31 00:51:42.868209 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.868172 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cn4s\" (UniqueName: \"kubernetes.io/projected/e8f84625-11a2-4de1-8d23-45555705aedd-kube-api-access-9cn4s\") pod \"proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl\" (UID: \"e8f84625-11a2-4de1-8d23-45555705aedd\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl" Oct 31 00:51:42.882878 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.882846 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cn4s\" (UniqueName: \"kubernetes.io/projected/e8f84625-11a2-4de1-8d23-45555705aedd-kube-api-access-9cn4s\") pod \"proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl\" (UID: \"e8f84625-11a2-4de1-8d23-45555705aedd\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl" Oct 31 00:51:42.979655 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:42.979589 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl" Oct 31 00:51:43.255033 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.254854 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl"] Oct 31 00:51:43.257990 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:51:43.257957 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f84625_11a2_4de1_8d23_45555705aedd.slice/crio-4c7cb398ba38c67c73bceeae0c92e4007a903ce88767138d337115c356573124 WatchSource:0}: Error finding container 4c7cb398ba38c67c73bceeae0c92e4007a903ce88767138d337115c356573124: Status 404 returned error can't find the container with id 4c7cb398ba38c67c73bceeae0c92e4007a903ce88767138d337115c356573124 Oct 31 00:51:43.260892 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.260875 5939 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:51:43.283808 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.283787 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht"] Oct 31 00:51:43.288415 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.288398 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:43.294167 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.294148 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 00:51:43.295579 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.295322 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:51:43.295579 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.295331 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 00:51:43.295579 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.295353 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 00:51:43.295579 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.295391 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\"" Oct 31 00:51:43.295579 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.295435 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\"" Oct 31 00:51:43.295579 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.295443 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 00:51:43.301579 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.301547 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht"] Oct 31 00:51:43.371941 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.371906 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e90e9252-0f94-418d-b629-352d3db06ed9-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:43.372143 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.371946 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7crz\" (UniqueName: \"kubernetes.io/projected/e90e9252-0f94-418d-b629-352d3db06ed9-kube-api-access-d7crz\") pod \"isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:43.372143 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.372097 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e90e9252-0f94-418d-b629-352d3db06ed9-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:43.372248 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.372141 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90e9252-0f94-418d-b629-352d3db06ed9-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:43.372248 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.372180 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e90e9252-0f94-418d-b629-352d3db06ed9-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:43.473043 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.473000 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e90e9252-0f94-418d-b629-352d3db06ed9-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:43.473043 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.473045 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90e9252-0f94-418d-b629-352d3db06ed9-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:43.473312 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.473066 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e90e9252-0f94-418d-b629-352d3db06ed9-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:43.473312 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.473101 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e90e9252-0f94-418d-b629-352d3db06ed9-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:43.473312 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.473122 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7crz\" (UniqueName: \"kubernetes.io/projected/e90e9252-0f94-418d-b629-352d3db06ed9-kube-api-access-d7crz\") pod \"isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:43.473538 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.473506 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90e9252-0f94-418d-b629-352d3db06ed9-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:43.473829 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.473805 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e90e9252-0f94-418d-b629-352d3db06ed9-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:43.473911 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.473838 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e90e9252-0f94-418d-b629-352d3db06ed9-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:43.475556 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.475532 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e90e9252-0f94-418d-b629-352d3db06ed9-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:43.487556 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.487535 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7crz\" (UniqueName: \"kubernetes.io/projected/e90e9252-0f94-418d-b629-352d3db06ed9-kube-api-access-d7crz\") pod \"isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:43.606572 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.606485 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:43.905733 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:43.905707 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht"] Oct 31 00:51:43.908525 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:51:43.908495 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90e9252_0f94_418d_b629_352d3db06ed9.slice/crio-1d79b899de00ef7c786c7e9049521a8db5a9c3a5e98de92352f23ae7d2407863 WatchSource:0}: Error finding container 1d79b899de00ef7c786c7e9049521a8db5a9c3a5e98de92352f23ae7d2407863: Status 404 returned error can't find the container with id 1d79b899de00ef7c786c7e9049521a8db5a9c3a5e98de92352f23ae7d2407863 Oct 31 00:51:44.261727 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:44.261590 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl" event={"ID":"e8f84625-11a2-4de1-8d23-45555705aedd","Type":"ContainerStarted","Data":"4c7cb398ba38c67c73bceeae0c92e4007a903ce88767138d337115c356573124"} Oct 31 00:51:44.263402 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:44.263368 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" event={"ID":"e90e9252-0f94-418d-b629-352d3db06ed9","Type":"ContainerStarted","Data":"0e9b60587e9e754889ae646f9457f9edd0c912f9c56d235173695f2ccee2b58d"} Oct 31 00:51:44.263533 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:44.263434 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" event={"ID":"e90e9252-0f94-418d-b629-352d3db06ed9","Type":"ContainerStarted","Data":"1d79b899de00ef7c786c7e9049521a8db5a9c3a5e98de92352f23ae7d2407863"} Oct 31 00:51:46.275620 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:46.275586 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl" event={"ID":"e8f84625-11a2-4de1-8d23-45555705aedd","Type":"ContainerStarted","Data":"886c034c4680d07edd2d9e122dc6cf6ffb7bcf40caf72e22868c4924593a06dc"} Oct 31 00:51:46.276117 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:46.275786 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl" Oct 31 00:51:46.277761 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:46.277743 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl" Oct 31 00:51:46.297527 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:46.297484 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl" podStartSLOduration=2.164629411 podStartE2EDuration="4.297467461s" totalImagesPullingTime="2.13283805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:46.297171848 +0000 UTC m=+1318.786447027" watchObservedRunningTime="2026-10-31 00:51:46.297467461 +0000 UTC m=+1318.786742641" Oct 31 00:51:48.287488 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:48.287451 5939 generic.go:356] "Generic (PLEG): container finished" podID="e90e9252-0f94-418d-b629-352d3db06ed9" containerID="0e9b60587e9e754889ae646f9457f9edd0c912f9c56d235173695f2ccee2b58d" exitCode=0 Oct 31 00:51:48.288270 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:48.288244 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" event={"ID":"e90e9252-0f94-418d-b629-352d3db06ed9","Type":"ContainerDied","Data":"0e9b60587e9e754889ae646f9457f9edd0c912f9c56d235173695f2ccee2b58d"} Oct 31 00:51:49.294286 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:49.294254 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" event={"ID":"e90e9252-0f94-418d-b629-352d3db06ed9","Type":"ContainerStarted","Data":"465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0"} Oct 31 00:51:49.294811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:49.294312 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" event={"ID":"e90e9252-0f94-418d-b629-352d3db06ed9","Type":"ContainerStarted","Data":"93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928"} Oct 31 00:51:49.294811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:49.294502 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:49.325392 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:49.325350 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" podStartSLOduration=2.181603935 podStartE2EDuration="6.325336017s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.143732082s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:49.323712865 +0000 UTC m=+1321.812988047" watchObservedRunningTime="2026-10-31 00:51:49.325336017 +0000 UTC m=+1321.814611193" Oct 31 00:51:50.098073 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:50.098033 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp"] Oct 31 00:51:50.101143 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:50.101120 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:50.107264 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:50.107236 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-0e21ef4e-kube-rbac-proxy-sar-config\"" Oct 31 00:51:50.107578 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:50.107554 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-0e21ef4e-dockercfg-2wdgc\"" Oct 31 00:51:50.107578 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:50.107554 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"proxy-routing-0e21ef4e-predictor-serving-cert\"" Oct 31 00:51:50.114851 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:50.114829 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp"] Oct 31 00:51:50.232829 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:50.232790 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94e6e6be-c8cf-4044-a027-8e29eaede391-kube-rbac-proxy-sar-config\") pod \"proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:50.233035 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:50.232846 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l26db\" (UniqueName: \"kubernetes.io/projected/94e6e6be-c8cf-4044-a027-8e29eaede391-kube-api-access-l26db\") pod \"proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:50.233035 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:50.232947 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94e6e6be-c8cf-4044-a027-8e29eaede391-proxy-tls\") pod \"proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:50.298361 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:50.298332 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:50.334017 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:50.333981 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94e6e6be-c8cf-4044-a027-8e29eaede391-kube-rbac-proxy-sar-config\") pod \"proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:50.334214 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:50.334059 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l26db\" (UniqueName: \"kubernetes.io/projected/94e6e6be-c8cf-4044-a027-8e29eaede391-kube-api-access-l26db\") pod \"proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:50.334214 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:50.334083 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94e6e6be-c8cf-4044-a027-8e29eaede391-proxy-tls\") pod \"proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:50.334903 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:50.334877 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94e6e6be-c8cf-4044-a027-8e29eaede391-kube-rbac-proxy-sar-config\") pod \"proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:50.336658 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:50.336640 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94e6e6be-c8cf-4044-a027-8e29eaede391-proxy-tls\") pod \"proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:50.348740 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:50.348680 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l26db\" (UniqueName: \"kubernetes.io/projected/94e6e6be-c8cf-4044-a027-8e29eaede391-kube-api-access-l26db\") pod \"proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:56.307827 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:56.307799 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:51:56.488536 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:56.488502 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-15d6681b-85cb-48db-836d-919ed7bc2c32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81\") pod \"proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:56.497733 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:56.497709 5939 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-15d6681b-85cb-48db-836d-919ed7bc2c32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81\") pod \"proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") device path: \"\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:56.589377 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:56.589304 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-15d6681b-85cb-48db-836d-919ed7bc2c32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81\") pod \"proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:56.589496 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:56.589401 5939 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-15d6681b-85cb-48db-836d-919ed7bc2c32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81\") pod \"proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") DevicePath \"\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:56.594065 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:56.594048 5939 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-15d6681b-85cb-48db-836d-919ed7bc2c32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81\") pod \"proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") DevicePath \"csi-04d74cf946b08ebd8bafd32b63259630b25bc98904a0b26b1c7728f22278636c\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:56.768614 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:56.768581 5939 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-15d6681b-85cb-48db-836d-919ed7bc2c32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81\") pod \"proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/dbae13b7beea67d74f2c746fa717735d5ee712062fcc5c5847939b7343315370/globalmount\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:56.783226 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:56.783196 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-15d6681b-85cb-48db-836d-919ed7bc2c32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81\") pod \"proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:57.015808 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:57.015776 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:57.296906 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:57.296833 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp"] Oct 31 00:51:57.299915 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:51:57.299887 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e6e6be_c8cf_4044_a027_8e29eaede391.slice/crio-5d262be1f690cdb988e6973836ec8b67bf0c534db245ebe376fe75135ce72e83 WatchSource:0}: Error finding container 5d262be1f690cdb988e6973836ec8b67bf0c534db245ebe376fe75135ce72e83: Status 404 returned error can't find the container with id 5d262be1f690cdb988e6973836ec8b67bf0c534db245ebe376fe75135ce72e83 Oct 31 00:51:57.329150 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:57.329126 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" event={"ID":"94e6e6be-c8cf-4044-a027-8e29eaede391","Type":"ContainerStarted","Data":"5d262be1f690cdb988e6973836ec8b67bf0c534db245ebe376fe75135ce72e83"} Oct 31 00:51:58.335947 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:58.335910 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" event={"ID":"94e6e6be-c8cf-4044-a027-8e29eaede391","Type":"ContainerStarted","Data":"3ac6f48d0930baa11f0d63949378b9c1adf8af5f89039f36c3b2191c9a4f7a26"} Oct 31 00:51:58.336430 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:58.335978 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" event={"ID":"94e6e6be-c8cf-4044-a027-8e29eaede391","Type":"ContainerStarted","Data":"e05ec6d7151df08c33f981993534ba0cba7c923e2d8e3b0faab40babe3e44531"} Oct 31 00:51:58.336430 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:58.336016 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:58.336430 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:58.336044 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:58.338281 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:58.338257 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:51:58.364630 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:51:58.364577 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" podStartSLOduration=11.364565376 podStartE2EDuration="11.364565376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:58.362153956 +0000 UTC m=+1330.851429135" watchObservedRunningTime="2026-10-31 00:51:58.364565376 +0000 UTC m=+1330.853840552" Oct 31 00:52:00.509188 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:52:00.509151 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:52:05.347649 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:05.347621 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:52:07.206941 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.206901 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb"] Oct 31 00:52:07.210286 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.210262 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:07.224407 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.224369 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb"] Oct 31 00:52:07.282720 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.282696 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-proxy-tls\") pod \"proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb\" (UID: \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:07.282871 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.282744 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-15d6681b-85cb-48db-836d-919ed7bc2c32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81\") pod \"proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb\" (UID: \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:07.282871 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.282831 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d69dc\" (UniqueName: \"kubernetes.io/projected/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-kube-api-access-d69dc\") pod \"proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb\" (UID: \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:07.282871 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.282840 5939 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-15d6681b-85cb-48db-836d-919ed7bc2c32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81\") pod \"proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb\" (UID: \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\") DevicePath \"csi-04d74cf946b08ebd8bafd32b63259630b25bc98904a0b26b1c7728f22278636c\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:07.282871 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.282858 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-kube-rbac-proxy-sar-config\") pod \"proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb\" (UID: \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:07.287369 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.287352 5939 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-15d6681b-85cb-48db-836d-919ed7bc2c32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81\") pod \"proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb\" (UID: \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\") DevicePath \"csi-04d74cf946b08ebd8bafd32b63259630b25bc98904a0b26b1c7728f22278636c\"" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:07.302004 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.301980 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-15d6681b-85cb-48db-836d-919ed7bc2c32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81\") pod \"proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb\" (UID: \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:07.383426 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.383402 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-proxy-tls\") pod \"proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb\" (UID: \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:07.383567 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.383484 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d69dc\" (UniqueName: \"kubernetes.io/projected/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-kube-api-access-d69dc\") pod \"proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb\" (UID: \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:07.383567 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.383515 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-kube-rbac-proxy-sar-config\") pod \"proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb\" (UID: \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:07.384206 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.384183 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-kube-rbac-proxy-sar-config\") pod \"proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb\" (UID: \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:07.385837 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.385813 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-proxy-tls\") pod \"proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb\" (UID: \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:07.397914 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.397893 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d69dc\" (UniqueName: \"kubernetes.io/projected/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-kube-api-access-d69dc\") pod \"proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb\" (UID: \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\") " pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:07.533179 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.533088 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:07.813608 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:07.813526 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb"] Oct 31 00:52:07.816415 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:52:07.816390 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09422611_0fe3_4cf1_8f0c_bba1b9ee1a24.slice/crio-cdc12c8dc20750b72aed24bddfce553862a8ae2ac95d00b0d4abfe96442706fa WatchSource:0}: Error finding container cdc12c8dc20750b72aed24bddfce553862a8ae2ac95d00b0d4abfe96442706fa: Status 404 returned error can't find the container with id cdc12c8dc20750b72aed24bddfce553862a8ae2ac95d00b0d4abfe96442706fa Oct 31 00:52:08.378448 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:08.378414 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" event={"ID":"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24","Type":"ContainerStarted","Data":"0ce3d648cfdb3481c999ad5aae787602ca6423c1ce17301220211c97cc73e4e2"} Oct 31 00:52:08.378973 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:08.378484 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:08.378973 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:08.378514 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:08.378973 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:08.378536 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" event={"ID":"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24","Type":"ContainerStarted","Data":"15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9"} Oct 31 00:52:08.378973 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:08.378561 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" event={"ID":"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24","Type":"ContainerStarted","Data":"cdc12c8dc20750b72aed24bddfce553862a8ae2ac95d00b0d4abfe96442706fa"} Oct 31 00:52:08.380785 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:08.380762 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:08.403993 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:08.403955 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" podStartSLOduration=1.4039445210000001 podStartE2EDuration="1.403944521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:08.403051654 +0000 UTC m=+1340.892326834" watchObservedRunningTime="2026-10-31 00:52:08.403944521 +0000 UTC m=+1340.893219699" Oct 31 00:52:15.391659 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:15.391629 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:52:15.455885 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:15.455852 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp"] Oct 31 00:52:15.456211 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:15.456148 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" podUID="94e6e6be-c8cf-4044-a027-8e29eaede391" containerName="kserve-container" containerID="cri-o://e05ec6d7151df08c33f981993534ba0cba7c923e2d8e3b0faab40babe3e44531" gracePeriod=30 Oct 31 00:52:15.456382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:15.456175 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" podUID="94e6e6be-c8cf-4044-a027-8e29eaede391" containerName="kube-rbac-proxy" containerID="cri-o://3ac6f48d0930baa11f0d63949378b9c1adf8af5f89039f36c3b2191c9a4f7a26" gracePeriod=30 Oct 31 00:52:16.428489 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:16.428455 5939 generic.go:356] "Generic (PLEG): container finished" podID="94e6e6be-c8cf-4044-a027-8e29eaede391" containerID="3ac6f48d0930baa11f0d63949378b9c1adf8af5f89039f36c3b2191c9a4f7a26" exitCode=0 Oct 31 00:52:16.428903 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:16.428539 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" event={"ID":"94e6e6be-c8cf-4044-a027-8e29eaede391","Type":"ContainerDied","Data":"3ac6f48d0930baa11f0d63949378b9c1adf8af5f89039f36c3b2191c9a4f7a26"} Oct 31 00:52:20.344345 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:20.344295 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" podUID="94e6e6be-c8cf-4044-a027-8e29eaede391" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused" Oct 31 00:52:25.344410 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:25.344367 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" podUID="94e6e6be-c8cf-4044-a027-8e29eaede391" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused" Oct 31 00:52:26.367738 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:26.367692 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:52:30.344620 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:30.344539 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" podUID="94e6e6be-c8cf-4044-a027-8e29eaede391" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused" Oct 31 00:52:30.345003 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:30.344663 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:52:30.509066 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:52:30.509034 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:52:33.472895 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:33.472863 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht"] Oct 31 00:52:33.473356 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:33.473192 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" podUID="e90e9252-0f94-418d-b629-352d3db06ed9" containerName="kserve-container" containerID="cri-o://93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928" gracePeriod=30 Oct 31 00:52:33.473356 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:33.473254 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" podUID="e90e9252-0f94-418d-b629-352d3db06ed9" containerName="kube-rbac-proxy" containerID="cri-o://465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0" gracePeriod=30 Oct 31 00:52:34.117770 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.117746 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:52:34.231515 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.231443 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90e9252-0f94-418d-b629-352d3db06ed9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e90e9252-0f94-418d-b629-352d3db06ed9-kube-rbac-proxy-sar-config\") pod \"e90e9252-0f94-418d-b629-352d3db06ed9\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " Oct 31 00:52:34.231515 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.231488 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90e9252-0f94-418d-b629-352d3db06ed9-kube-api-access-d7crz\" (UniqueName: \"kubernetes.io/projected/e90e9252-0f94-418d-b629-352d3db06ed9-kube-api-access-d7crz\") pod \"e90e9252-0f94-418d-b629-352d3db06ed9\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " Oct 31 00:52:34.231767 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.231535 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90e9252-0f94-418d-b629-352d3db06ed9-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e90e9252-0f94-418d-b629-352d3db06ed9-cabundle-cert\") pod \"e90e9252-0f94-418d-b629-352d3db06ed9\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " Oct 31 00:52:34.231767 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.231569 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90e9252-0f94-418d-b629-352d3db06ed9-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90e9252-0f94-418d-b629-352d3db06ed9-kserve-provision-location\") pod \"e90e9252-0f94-418d-b629-352d3db06ed9\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " Oct 31 00:52:34.231767 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.231607 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e90e9252-0f94-418d-b629-352d3db06ed9-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e90e9252-0f94-418d-b629-352d3db06ed9-proxy-tls\") pod \"e90e9252-0f94-418d-b629-352d3db06ed9\" (UID: \"e90e9252-0f94-418d-b629-352d3db06ed9\") " Oct 31 00:52:34.231929 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.231905 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90e9252-0f94-418d-b629-352d3db06ed9-kube-rbac-proxy-sar-config" pod "e90e9252-0f94-418d-b629-352d3db06ed9" (UID: "e90e9252-0f94-418d-b629-352d3db06ed9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:34.231988 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.231945 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90e9252-0f94-418d-b629-352d3db06ed9-kserve-provision-location" pod "e90e9252-0f94-418d-b629-352d3db06ed9" (UID: "e90e9252-0f94-418d-b629-352d3db06ed9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:34.232040 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.231991 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90e9252-0f94-418d-b629-352d3db06ed9-cabundle-cert" pod "e90e9252-0f94-418d-b629-352d3db06ed9" (UID: "e90e9252-0f94-418d-b629-352d3db06ed9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:34.233763 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.233735 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90e9252-0f94-418d-b629-352d3db06ed9-kube-api-access-d7crz" pod "e90e9252-0f94-418d-b629-352d3db06ed9" (UID: "e90e9252-0f94-418d-b629-352d3db06ed9"). InnerVolumeSpecName "kube-api-access-d7crz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:34.233864 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.233803 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90e9252-0f94-418d-b629-352d3db06ed9-proxy-tls" pod "e90e9252-0f94-418d-b629-352d3db06ed9" (UID: "e90e9252-0f94-418d-b629-352d3db06ed9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:52:34.332935 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.332904 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e90e9252-0f94-418d-b629-352d3db06ed9-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:34.332935 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.332932 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e90e9252-0f94-418d-b629-352d3db06ed9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:34.333091 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.332942 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7crz\" (UniqueName: \"kubernetes.io/projected/e90e9252-0f94-418d-b629-352d3db06ed9-kube-api-access-d7crz\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:34.333091 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.332952 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e90e9252-0f94-418d-b629-352d3db06ed9-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:34.333091 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.332961 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90e9252-0f94-418d-b629-352d3db06ed9-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:34.508461 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.508380 5939 generic.go:356] "Generic (PLEG): container finished" podID="e90e9252-0f94-418d-b629-352d3db06ed9" containerID="465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0" exitCode=0 Oct 31 00:52:34.508461 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.508404 5939 generic.go:356] "Generic (PLEG): container finished" podID="e90e9252-0f94-418d-b629-352d3db06ed9" containerID="93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928" exitCode=0 Oct 31 00:52:34.508461 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.508452 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" event={"ID":"e90e9252-0f94-418d-b629-352d3db06ed9","Type":"ContainerDied","Data":"465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0"} Oct 31 00:52:34.508461 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.508453 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" Oct 31 00:52:34.509088 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.508499 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" event={"ID":"e90e9252-0f94-418d-b629-352d3db06ed9","Type":"ContainerDied","Data":"93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928"} Oct 31 00:52:34.509088 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.508518 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht" event={"ID":"e90e9252-0f94-418d-b629-352d3db06ed9","Type":"ContainerDied","Data":"1d79b899de00ef7c786c7e9049521a8db5a9c3a5e98de92352f23ae7d2407863"} Oct 31 00:52:34.509088 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.508541 5939 scope.go:122] "RemoveContainer" containerID="465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0" Oct 31 00:52:34.517818 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.517797 5939 scope.go:122] "RemoveContainer" containerID="93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928" Oct 31 00:52:34.525103 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.525083 5939 scope.go:122] "RemoveContainer" containerID="0e9b60587e9e754889ae646f9457f9edd0c912f9c56d235173695f2ccee2b58d" Oct 31 00:52:34.532655 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.532641 5939 scope.go:122] "RemoveContainer" containerID="465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0" Oct 31 00:52:34.533009 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:52:34.532982 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0\": container with ID starting with 465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0 not found: ID does not exist" containerID="465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0" Oct 31 00:52:34.533092 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.533022 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0"} err="failed to get container status \"465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0\": rpc error: code = NotFound desc = could not find container \"465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0\": container with ID starting with 465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0 not found: ID does not exist" Oct 31 00:52:34.533092 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.533047 5939 scope.go:122] "RemoveContainer" containerID="93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928" Oct 31 00:52:34.533433 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:52:34.533405 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928\": container with ID starting with 93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928 not found: ID does not exist" containerID="93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928" Oct 31 00:52:34.533512 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.533437 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928"} err="failed to get container status \"93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928\": rpc error: code = NotFound desc = could not find container \"93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928\": container with ID starting with 93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928 not found: ID does not exist" Oct 31 00:52:34.533512 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.533461 5939 scope.go:122] "RemoveContainer" containerID="0e9b60587e9e754889ae646f9457f9edd0c912f9c56d235173695f2ccee2b58d" Oct 31 00:52:34.533910 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:52:34.533815 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e9b60587e9e754889ae646f9457f9edd0c912f9c56d235173695f2ccee2b58d\": container with ID starting with 0e9b60587e9e754889ae646f9457f9edd0c912f9c56d235173695f2ccee2b58d not found: ID does not exist" containerID="0e9b60587e9e754889ae646f9457f9edd0c912f9c56d235173695f2ccee2b58d" Oct 31 00:52:34.533910 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.533844 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e9b60587e9e754889ae646f9457f9edd0c912f9c56d235173695f2ccee2b58d"} err="failed to get container status \"0e9b60587e9e754889ae646f9457f9edd0c912f9c56d235173695f2ccee2b58d\": rpc error: code = NotFound desc = could not find container \"0e9b60587e9e754889ae646f9457f9edd0c912f9c56d235173695f2ccee2b58d\": container with ID starting with 0e9b60587e9e754889ae646f9457f9edd0c912f9c56d235173695f2ccee2b58d not found: ID does not exist" Oct 31 00:52:34.533910 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.533860 5939 scope.go:122] "RemoveContainer" containerID="465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0" Oct 31 00:52:34.534351 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.534330 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0"} err="failed to get container status \"465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0\": rpc error: code = NotFound desc = could not find container \"465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0\": container with ID starting with 465fde02987b69686ccf129dfbc3e3b59c05b27b5fdcfd932589ce546f72d1e0 not found: ID does not exist" Oct 31 00:52:34.534351 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.534349 5939 scope.go:122] "RemoveContainer" containerID="93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928" Oct 31 00:52:34.534594 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.534574 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928"} err="failed to get container status \"93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928\": rpc error: code = NotFound desc = could not find container \"93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928\": container with ID starting with 93315264ca1db9bd3c73b696f25712b9b30985657321cbce52ca993a4c172928 not found: ID does not exist" Oct 31 00:52:34.534656 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.534594 5939 scope.go:122] "RemoveContainer" containerID="0e9b60587e9e754889ae646f9457f9edd0c912f9c56d235173695f2ccee2b58d" Oct 31 00:52:34.534854 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.534830 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e9b60587e9e754889ae646f9457f9edd0c912f9c56d235173695f2ccee2b58d"} err="failed to get container status \"0e9b60587e9e754889ae646f9457f9edd0c912f9c56d235173695f2ccee2b58d\": rpc error: code = NotFound desc = could not find container \"0e9b60587e9e754889ae646f9457f9edd0c912f9c56d235173695f2ccee2b58d\": container with ID starting with 0e9b60587e9e754889ae646f9457f9edd0c912f9c56d235173695f2ccee2b58d not found: ID does not exist" Oct 31 00:52:34.535265 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.535248 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht"] Oct 31 00:52:34.540916 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:34.540895 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-predictive-lightgbm-v2-predictor-77858f544f-vzpht"] Oct 31 00:52:35.344708 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:35.344648 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" podUID="94e6e6be-c8cf-4044-a027-8e29eaede391" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused" Oct 31 00:52:35.653030 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:35.652999 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90e9252-0f94-418d-b629-352d3db06ed9" path="/var/lib/kubelet/pods/e90e9252-0f94-418d-b629-352d3db06ed9/volumes" Oct 31 00:52:40.088909 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:40.088877 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-70eec87d"] Oct 31 00:52:40.089288 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:40.089236 5939 cpu_manager.go:414] "containerMap: removing container" podUID="e90e9252-0f94-418d-b629-352d3db06ed9" containerName="kube-rbac-proxy" Oct 31 00:52:40.089288 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:40.089247 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90e9252-0f94-418d-b629-352d3db06ed9" containerName="kube-rbac-proxy" Oct 31 00:52:40.089288 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:40.089262 5939 cpu_manager.go:414] "containerMap: removing container" podUID="e90e9252-0f94-418d-b629-352d3db06ed9" containerName="storage-initializer" Oct 31 00:52:40.089288 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:40.089268 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90e9252-0f94-418d-b629-352d3db06ed9" containerName="storage-initializer" Oct 31 00:52:40.089288 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:40.089289 5939 cpu_manager.go:414] "containerMap: removing container" podUID="e90e9252-0f94-418d-b629-352d3db06ed9" containerName="kserve-container" Oct 31 00:52:40.089442 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:40.089294 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90e9252-0f94-418d-b629-352d3db06ed9" containerName="kserve-container" Oct 31 00:52:40.089442 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:40.089351 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90e9252-0f94-418d-b629-352d3db06ed9" containerName="kube-rbac-proxy" Oct 31 00:52:40.089442 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:40.089363 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90e9252-0f94-418d-b629-352d3db06ed9" containerName="kserve-container" Oct 31 00:52:40.092417 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:40.092401 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-70eec87d" Oct 31 00:52:40.101526 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:40.101490 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-70eec87d"] Oct 31 00:52:40.185859 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:40.185831 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8vq9\" (UniqueName: \"kubernetes.io/projected/c70af9db-119b-4e71-9b9f-3960d2287f14-kube-api-access-t8vq9\") pod \"curl-metrics-70eec87d\" (UID: \"c70af9db-119b-4e71-9b9f-3960d2287f14\") " pod="kserve/curl-metrics-70eec87d" Oct 31 00:52:40.286717 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:40.286684 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8vq9\" (UniqueName: \"kubernetes.io/projected/c70af9db-119b-4e71-9b9f-3960d2287f14-kube-api-access-t8vq9\") pod \"curl-metrics-70eec87d\" (UID: \"c70af9db-119b-4e71-9b9f-3960d2287f14\") " pod="kserve/curl-metrics-70eec87d" Oct 31 00:52:40.301531 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:40.301498 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8vq9\" (UniqueName: \"kubernetes.io/projected/c70af9db-119b-4e71-9b9f-3960d2287f14-kube-api-access-t8vq9\") pod \"curl-metrics-70eec87d\" (UID: \"c70af9db-119b-4e71-9b9f-3960d2287f14\") " pod="kserve/curl-metrics-70eec87d" Oct 31 00:52:40.344706 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:40.344629 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" podUID="94e6e6be-c8cf-4044-a027-8e29eaede391" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused" Oct 31 00:52:40.411687 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:40.411635 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-70eec87d" Oct 31 00:52:40.686122 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:40.685983 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-70eec87d"] Oct 31 00:52:40.688790 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:52:40.688763 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc70af9db_119b_4e71_9b9f_3960d2287f14.slice/crio-6e8345ccc347ae67f26b90bdfb880b6b14534cb12831dbddaf04330477aefa9a WatchSource:0}: Error finding container 6e8345ccc347ae67f26b90bdfb880b6b14534cb12831dbddaf04330477aefa9a: Status 404 returned error can't find the container with id 6e8345ccc347ae67f26b90bdfb880b6b14534cb12831dbddaf04330477aefa9a Oct 31 00:52:41.540580 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:41.540550 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-70eec87d" event={"ID":"c70af9db-119b-4e71-9b9f-3960d2287f14","Type":"ContainerStarted","Data":"6e8345ccc347ae67f26b90bdfb880b6b14534cb12831dbddaf04330477aefa9a"} Oct 31 00:52:42.546201 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:42.546161 5939 generic.go:356] "Generic (PLEG): container finished" podID="c70af9db-119b-4e71-9b9f-3960d2287f14" containerID="d2e740f7b6ade9f8a3cb5977ac87575ef4425b8a484249b60ddd64ca3aaff55f" exitCode=0 Oct 31 00:52:42.546563 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:42.546222 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-70eec87d" event={"ID":"c70af9db-119b-4e71-9b9f-3960d2287f14","Type":"ContainerDied","Data":"d2e740f7b6ade9f8a3cb5977ac87575ef4425b8a484249b60ddd64ca3aaff55f"} Oct 31 00:52:43.676724 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:43.676701 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-70eec87d" Oct 31 00:52:43.748051 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:43.748017 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-70eec87d_c70af9db-119b-4e71-9b9f-3960d2287f14/curl-metrics-70eec87d/0.log" Oct 31 00:52:43.817373 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:43.817304 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70af9db-119b-4e71-9b9f-3960d2287f14-kube-api-access-t8vq9\" (UniqueName: \"kubernetes.io/projected/c70af9db-119b-4e71-9b9f-3960d2287f14-kube-api-access-t8vq9\") pod \"c70af9db-119b-4e71-9b9f-3960d2287f14\" (UID: \"c70af9db-119b-4e71-9b9f-3960d2287f14\") " Oct 31 00:52:43.819756 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:43.819728 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70af9db-119b-4e71-9b9f-3960d2287f14-kube-api-access-t8vq9" pod "c70af9db-119b-4e71-9b9f-3960d2287f14" (UID: "c70af9db-119b-4e71-9b9f-3960d2287f14"). InnerVolumeSpecName "kube-api-access-t8vq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:43.820063 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:43.820042 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-70eec87d"] Oct 31 00:52:43.826248 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:43.826229 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-70eec87d"] Oct 31 00:52:43.918846 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:43.918819 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8vq9\" (UniqueName: \"kubernetes.io/projected/c70af9db-119b-4e71-9b9f-3960d2287f14-kube-api-access-t8vq9\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:44.441624 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:44.441586 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-aa18dfc6"] Oct 31 00:52:44.442063 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:44.442044 5939 cpu_manager.go:414] "containerMap: removing container" podUID="c70af9db-119b-4e71-9b9f-3960d2287f14" containerName="curl-metrics-70eec87d" Oct 31 00:52:44.442148 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:44.442065 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70af9db-119b-4e71-9b9f-3960d2287f14" containerName="curl-metrics-70eec87d" Oct 31 00:52:44.442200 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:44.442186 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70af9db-119b-4e71-9b9f-3960d2287f14" containerName="curl-metrics-70eec87d" Oct 31 00:52:44.445289 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:44.445268 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-aa18dfc6" Oct 31 00:52:44.454349 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:44.454328 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-aa18dfc6"] Oct 31 00:52:44.523812 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:44.523775 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79q5h\" (UniqueName: \"kubernetes.io/projected/5a4896a9-9e03-4ff5-8adb-1e685f0160dc-kube-api-access-79q5h\") pod \"curl-metrics-aa18dfc6\" (UID: \"5a4896a9-9e03-4ff5-8adb-1e685f0160dc\") " pod="kserve/curl-metrics-aa18dfc6" Oct 31 00:52:44.557111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:44.557086 5939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e8345ccc347ae67f26b90bdfb880b6b14534cb12831dbddaf04330477aefa9a" Oct 31 00:52:44.557111 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:44.557089 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-70eec87d" Oct 31 00:52:44.567816 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:52:44.567791 5939 status_manager.go:1045] "Failed to get status for pod" err="pods \"curl-metrics-70eec87d\" is forbidden: User \"system:node:ip-10-0-143-213.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-143-213.us-east-2.compute.internal' and this object" podUID="c70af9db-119b-4e71-9b9f-3960d2287f14" pod="kserve/curl-metrics-70eec87d" Oct 31 00:52:44.625059 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:44.625034 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79q5h\" (UniqueName: \"kubernetes.io/projected/5a4896a9-9e03-4ff5-8adb-1e685f0160dc-kube-api-access-79q5h\") pod \"curl-metrics-aa18dfc6\" (UID: \"5a4896a9-9e03-4ff5-8adb-1e685f0160dc\") " pod="kserve/curl-metrics-aa18dfc6" Oct 31 00:52:44.639920 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:44.639895 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79q5h\" (UniqueName: \"kubernetes.io/projected/5a4896a9-9e03-4ff5-8adb-1e685f0160dc-kube-api-access-79q5h\") pod \"curl-metrics-aa18dfc6\" (UID: \"5a4896a9-9e03-4ff5-8adb-1e685f0160dc\") " pod="kserve/curl-metrics-aa18dfc6" Oct 31 00:52:44.757067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:44.756996 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-aa18dfc6" Oct 31 00:52:45.030836 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.030761 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-aa18dfc6"] Oct 31 00:52:45.033983 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:52:45.033958 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a4896a9_9e03_4ff5_8adb_1e685f0160dc.slice/crio-df5764c28bd7791d0876b05734be603dd1d3698c329681105c73471d3a0dd059 WatchSource:0}: Error finding container df5764c28bd7791d0876b05734be603dd1d3698c329681105c73471d3a0dd059: Status 404 returned error can't find the container with id df5764c28bd7791d0876b05734be603dd1d3698c329681105c73471d3a0dd059 Oct 31 00:52:45.344456 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.344365 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" podUID="94e6e6be-c8cf-4044-a027-8e29eaede391" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused" Oct 31 00:52:45.563479 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.563328 5939 generic.go:356] "Generic (PLEG): container finished" podID="94e6e6be-c8cf-4044-a027-8e29eaede391" containerID="e05ec6d7151df08c33f981993534ba0cba7c923e2d8e3b0faab40babe3e44531" exitCode=137 Oct 31 00:52:45.563479 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.563418 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" event={"ID":"94e6e6be-c8cf-4044-a027-8e29eaede391","Type":"ContainerDied","Data":"e05ec6d7151df08c33f981993534ba0cba7c923e2d8e3b0faab40babe3e44531"} Oct 31 00:52:45.565685 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.565644 5939 generic.go:356] "Generic (PLEG): container finished" podID="5a4896a9-9e03-4ff5-8adb-1e685f0160dc" containerID="8aa02c90926ede54901c9c3eadd9747a365f3232b402f928f59a9f8a3062fe65" exitCode=0 Oct 31 00:52:45.565806 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.565708 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-aa18dfc6" event={"ID":"5a4896a9-9e03-4ff5-8adb-1e685f0160dc","Type":"ContainerDied","Data":"8aa02c90926ede54901c9c3eadd9747a365f3232b402f928f59a9f8a3062fe65"} Oct 31 00:52:45.565806 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.565747 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-aa18dfc6" event={"ID":"5a4896a9-9e03-4ff5-8adb-1e685f0160dc","Type":"ContainerStarted","Data":"df5764c28bd7791d0876b05734be603dd1d3698c329681105c73471d3a0dd059"} Oct 31 00:52:45.568713 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:52:45.568691 5939 status_manager.go:1045] "Failed to get status for pod" err="pods \"curl-metrics-70eec87d\" is forbidden: User \"system:node:ip-10-0-143-213.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-143-213.us-east-2.compute.internal' and this object" podUID="c70af9db-119b-4e71-9b9f-3960d2287f14" pod="kserve/curl-metrics-70eec87d" Oct 31 00:52:45.625923 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.625902 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:52:45.628750 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:52:45.628725 5939 status_manager.go:1045] "Failed to get status for pod" err="pods \"curl-metrics-70eec87d\" is forbidden: User \"system:node:ip-10-0-143-213.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-143-213.us-east-2.compute.internal' and this object" podUID="c70af9db-119b-4e71-9b9f-3960d2287f14" pod="kserve/curl-metrics-70eec87d" Oct 31 00:52:45.652975 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.652949 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c70af9db-119b-4e71-9b9f-3960d2287f14" path="/var/lib/kubelet/pods/c70af9db-119b-4e71-9b9f-3960d2287f14/volumes" Oct 31 00:52:45.736514 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.736484 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e6e6be-c8cf-4044-a027-8e29eaede391-kube-api-access-l26db\" (UniqueName: \"kubernetes.io/projected/94e6e6be-c8cf-4044-a027-8e29eaede391-kube-api-access-l26db\") pod \"94e6e6be-c8cf-4044-a027-8e29eaede391\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") " Oct 31 00:52:45.736686 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.736556 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94e6e6be-c8cf-4044-a027-8e29eaede391-proxy-tls\" (UniqueName: \"kubernetes.io/secret/94e6e6be-c8cf-4044-a027-8e29eaede391-proxy-tls\") pod \"94e6e6be-c8cf-4044-a027-8e29eaede391\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") " Oct 31 00:52:45.736686 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.736588 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e6e6be-c8cf-4044-a027-8e29eaede391-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94e6e6be-c8cf-4044-a027-8e29eaede391-kube-rbac-proxy-sar-config\") pod \"94e6e6be-c8cf-4044-a027-8e29eaede391\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") " Oct 31 00:52:45.736776 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.736699 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81\") pod \"94e6e6be-c8cf-4044-a027-8e29eaede391\" (UID: \"94e6e6be-c8cf-4044-a027-8e29eaede391\") " Oct 31 00:52:45.736980 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.736953 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e6e6be-c8cf-4044-a027-8e29eaede391-kube-rbac-proxy-sar-config" pod "94e6e6be-c8cf-4044-a027-8e29eaede391" (UID: "94e6e6be-c8cf-4044-a027-8e29eaede391"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:45.738753 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.738732 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94e6e6be-c8cf-4044-a027-8e29eaede391-proxy-tls" pod "94e6e6be-c8cf-4044-a027-8e29eaede391" (UID: "94e6e6be-c8cf-4044-a027-8e29eaede391"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:52:45.738842 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.738815 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e6e6be-c8cf-4044-a027-8e29eaede391-kube-api-access-l26db" pod "94e6e6be-c8cf-4044-a027-8e29eaede391" (UID: "94e6e6be-c8cf-4044-a027-8e29eaede391"). InnerVolumeSpecName "kube-api-access-l26db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:45.740490 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.740467 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81" pod "94e6e6be-c8cf-4044-a027-8e29eaede391" (UID: "94e6e6be-c8cf-4044-a027-8e29eaede391"). InnerVolumeSpecName "pvc-15d6681b-85cb-48db-836d-919ed7bc2c32". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:52:45.837379 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.837354 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l26db\" (UniqueName: \"kubernetes.io/projected/94e6e6be-c8cf-4044-a027-8e29eaede391-kube-api-access-l26db\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:45.837379 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.837376 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94e6e6be-c8cf-4044-a027-8e29eaede391-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:45.837379 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:45.837385 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94e6e6be-c8cf-4044-a027-8e29eaede391-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:46.571967 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:46.571941 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" Oct 31 00:52:46.572145 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:46.571962 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp" event={"ID":"94e6e6be-c8cf-4044-a027-8e29eaede391","Type":"ContainerDied","Data":"5d262be1f690cdb988e6973836ec8b67bf0c534db245ebe376fe75135ce72e83"} Oct 31 00:52:46.572145 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:46.572044 5939 scope.go:122] "RemoveContainer" containerID="3ac6f48d0930baa11f0d63949378b9c1adf8af5f89039f36c3b2191c9a4f7a26" Oct 31 00:52:46.582024 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:46.582005 5939 scope.go:122] "RemoveContainer" containerID="e05ec6d7151df08c33f981993534ba0cba7c923e2d8e3b0faab40babe3e44531" Oct 31 00:52:46.599622 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:46.599596 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp"] Oct 31 00:52:46.605369 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:46.605325 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-6d7f96c4f6-bplhp"] Oct 31 00:52:46.699625 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:46.699599 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-aa18dfc6" Oct 31 00:52:46.738167 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:46.738140 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-aa18dfc6_5a4896a9-9e03-4ff5-8adb-1e685f0160dc/curl-metrics-aa18dfc6/0.log" Oct 31 00:52:46.807909 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:46.807884 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-aa18dfc6"] Oct 31 00:52:46.813720 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:46.813696 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-aa18dfc6"] Oct 31 00:52:46.847081 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:46.847032 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4896a9-9e03-4ff5-8adb-1e685f0160dc-kube-api-access-79q5h\" (UniqueName: \"kubernetes.io/projected/5a4896a9-9e03-4ff5-8adb-1e685f0160dc-kube-api-access-79q5h\") pod \"5a4896a9-9e03-4ff5-8adb-1e685f0160dc\" (UID: \"5a4896a9-9e03-4ff5-8adb-1e685f0160dc\") " Oct 31 00:52:46.849095 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:46.849070 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4896a9-9e03-4ff5-8adb-1e685f0160dc-kube-api-access-79q5h" pod "5a4896a9-9e03-4ff5-8adb-1e685f0160dc" (UID: "5a4896a9-9e03-4ff5-8adb-1e685f0160dc"). InnerVolumeSpecName "kube-api-access-79q5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:46.948192 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:46.948151 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79q5h\" (UniqueName: \"kubernetes.io/projected/5a4896a9-9e03-4ff5-8adb-1e685f0160dc-kube-api-access-79q5h\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:47.408059 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.408018 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g"] Oct 31 00:52:47.408447 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.408429 5939 cpu_manager.go:414] "containerMap: removing container" podUID="94e6e6be-c8cf-4044-a027-8e29eaede391" containerName="kserve-container" Oct 31 00:52:47.408539 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.408448 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e6e6be-c8cf-4044-a027-8e29eaede391" containerName="kserve-container" Oct 31 00:52:47.408539 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.408472 5939 cpu_manager.go:414] "containerMap: removing container" podUID="94e6e6be-c8cf-4044-a027-8e29eaede391" containerName="kube-rbac-proxy" Oct 31 00:52:47.408539 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.408479 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e6e6be-c8cf-4044-a027-8e29eaede391" containerName="kube-rbac-proxy" Oct 31 00:52:47.408539 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.408517 5939 cpu_manager.go:414] "containerMap: removing container" podUID="5a4896a9-9e03-4ff5-8adb-1e685f0160dc" containerName="curl-metrics-aa18dfc6" Oct 31 00:52:47.408539 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.408526 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4896a9-9e03-4ff5-8adb-1e685f0160dc" containerName="curl-metrics-aa18dfc6" Oct 31 00:52:47.408838 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.408625 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e6e6be-c8cf-4044-a027-8e29eaede391" containerName="kserve-container" Oct 31 00:52:47.408838 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.408638 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e6e6be-c8cf-4044-a027-8e29eaede391" containerName="kube-rbac-proxy" Oct 31 00:52:47.408838 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.408652 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4896a9-9e03-4ff5-8adb-1e685f0160dc" containerName="curl-metrics-aa18dfc6" Oct 31 00:52:47.411997 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.411977 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:47.418600 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.418570 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-predictor-serving-cert\"" Oct 31 00:52:47.418600 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.418586 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 00:52:47.418798 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.418690 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\"" Oct 31 00:52:47.418798 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.418709 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 00:52:47.418798 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.418746 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:52:47.418798 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.418778 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 00:52:47.419033 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.418835 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 00:52:47.425222 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.425194 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g"] Oct 31 00:52:47.551017 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.550988 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl"] Oct 31 00:52:47.551186 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.551160 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl" podUID="e8f84625-11a2-4de1-8d23-45555705aedd" containerName="kserve-container" containerID="cri-o://886c034c4680d07edd2d9e122dc6cf6ffb7bcf40caf72e22868c4924593a06dc" gracePeriod=30 Oct 31 00:52:47.554945 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.554925 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b17ead2-b3ae-485f-bd37-0e838656ca9e-proxy-tls\") pod \"isvc-sklearn-predictor-6fdbb7885f-jvx7g\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:47.555051 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.554959 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6fdbb7885f-jvx7g\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:47.555051 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.554986 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b17ead2-b3ae-485f-bd37-0e838656ca9e-cabundle-cert\") pod \"isvc-sklearn-predictor-6fdbb7885f-jvx7g\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:47.555134 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.555108 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khlxw\" (UniqueName: \"kubernetes.io/projected/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kube-api-access-khlxw\") pod \"isvc-sklearn-predictor-6fdbb7885f-jvx7g\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:47.555173 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.555134 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kserve-provision-location\") pod \"isvc-sklearn-predictor-6fdbb7885f-jvx7g\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:47.577776 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.577748 5939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df5764c28bd7791d0876b05734be603dd1d3698c329681105c73471d3a0dd059" Oct 31 00:52:47.577776 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.577772 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-aa18dfc6" Oct 31 00:52:47.588054 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:52:47.588028 5939 status_manager.go:1045] "Failed to get status for pod" err="pods \"curl-metrics-aa18dfc6\" is forbidden: User \"system:node:ip-10-0-143-213.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-143-213.us-east-2.compute.internal' and this object" podUID="5a4896a9-9e03-4ff5-8adb-1e685f0160dc" pod="kserve/curl-metrics-aa18dfc6" Oct 31 00:52:47.652844 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.652803 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4896a9-9e03-4ff5-8adb-1e685f0160dc" path="/var/lib/kubelet/pods/5a4896a9-9e03-4ff5-8adb-1e685f0160dc/volumes" Oct 31 00:52:47.652844 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:52:47.652809 5939 status_manager.go:1045] "Failed to get status for pod" err="pods \"curl-metrics-aa18dfc6\" is forbidden: User \"system:node:ip-10-0-143-213.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-143-213.us-east-2.compute.internal' and this object" podUID="5a4896a9-9e03-4ff5-8adb-1e685f0160dc" pod="kserve/curl-metrics-aa18dfc6" Oct 31 00:52:47.653374 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.653353 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e6e6be-c8cf-4044-a027-8e29eaede391" path="/var/lib/kubelet/pods/94e6e6be-c8cf-4044-a027-8e29eaede391/volumes" Oct 31 00:52:47.654205 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.654182 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb"] Oct 31 00:52:47.654384 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.654356 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" podUID="09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" containerName="kserve-container" containerID="cri-o://15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9" gracePeriod=30 Oct 31 00:52:47.654491 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.654378 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" podUID="09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" containerName="kube-rbac-proxy" containerID="cri-o://0ce3d648cfdb3481c999ad5aae787602ca6423c1ce17301220211c97cc73e4e2" gracePeriod=30 Oct 31 00:52:47.656364 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.656339 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khlxw\" (UniqueName: \"kubernetes.io/projected/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kube-api-access-khlxw\") pod \"isvc-sklearn-predictor-6fdbb7885f-jvx7g\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:47.656442 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.656384 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kserve-provision-location\") pod \"isvc-sklearn-predictor-6fdbb7885f-jvx7g\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:47.656515 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.656495 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b17ead2-b3ae-485f-bd37-0e838656ca9e-proxy-tls\") pod \"isvc-sklearn-predictor-6fdbb7885f-jvx7g\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:47.656575 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.656551 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6fdbb7885f-jvx7g\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:47.656627 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.656596 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b17ead2-b3ae-485f-bd37-0e838656ca9e-cabundle-cert\") pod \"isvc-sklearn-predictor-6fdbb7885f-jvx7g\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:47.656829 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.656813 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kserve-provision-location\") pod \"isvc-sklearn-predictor-6fdbb7885f-jvx7g\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:47.659570 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.659545 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\"" Oct 31 00:52:47.659643 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.659600 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:52:47.659643 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.659626 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-predictor-serving-cert\"" Oct 31 00:52:47.667480 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.667453 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b17ead2-b3ae-485f-bd37-0e838656ca9e-cabundle-cert\") pod \"isvc-sklearn-predictor-6fdbb7885f-jvx7g\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:47.667588 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.667570 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6fdbb7885f-jvx7g\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:47.669147 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.669127 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b17ead2-b3ae-485f-bd37-0e838656ca9e-proxy-tls\") pod \"isvc-sklearn-predictor-6fdbb7885f-jvx7g\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:47.671546 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.671525 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 00:52:47.681545 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.681528 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 00:52:47.690735 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.690717 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khlxw\" (UniqueName: \"kubernetes.io/projected/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kube-api-access-khlxw\") pod \"isvc-sklearn-predictor-6fdbb7885f-jvx7g\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:47.733360 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.733338 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 00:52:47.740655 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:47.740642 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:48.024291 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:48.024216 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g"] Oct 31 00:52:48.032408 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:48.032385 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 00:52:48.584449 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:48.584415 5939 generic.go:356] "Generic (PLEG): container finished" podID="09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" containerID="0ce3d648cfdb3481c999ad5aae787602ca6423c1ce17301220211c97cc73e4e2" exitCode=0 Oct 31 00:52:48.584638 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:48.584493 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" event={"ID":"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24","Type":"ContainerDied","Data":"0ce3d648cfdb3481c999ad5aae787602ca6423c1ce17301220211c97cc73e4e2"} Oct 31 00:52:48.586113 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:48.586091 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" event={"ID":"1b17ead2-b3ae-485f-bd37-0e838656ca9e","Type":"ContainerStarted","Data":"ae3b22272efb33fb54e033351aa6d6cd10d02c6233753c553ff0de30672e43d2"} Oct 31 00:52:48.586224 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:48.586136 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" event={"ID":"1b17ead2-b3ae-485f-bd37-0e838656ca9e","Type":"ContainerStarted","Data":"17547b2d64fed839299880422fec319d2613b242da75c7bd7c0d30f1c9fbd342"} Oct 31 00:52:50.388559 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:50.388518 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" podUID="09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused" Oct 31 00:52:52.604483 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:52.604400 5939 generic.go:356] "Generic (PLEG): container finished" podID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerID="ae3b22272efb33fb54e033351aa6d6cd10d02c6233753c553ff0de30672e43d2" exitCode=0 Oct 31 00:52:52.604483 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:52.604454 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" event={"ID":"1b17ead2-b3ae-485f-bd37-0e838656ca9e","Type":"ContainerDied","Data":"ae3b22272efb33fb54e033351aa6d6cd10d02c6233753c553ff0de30672e43d2"} Oct 31 00:52:53.610887 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:53.610851 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" event={"ID":"1b17ead2-b3ae-485f-bd37-0e838656ca9e","Type":"ContainerStarted","Data":"78d7637f6da9cb47b4f7060d45d18b0b13bfa77f7e8460e5a6c6db201d828479"} Oct 31 00:52:53.611294 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:53.610911 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" event={"ID":"1b17ead2-b3ae-485f-bd37-0e838656ca9e","Type":"ContainerStarted","Data":"7b14998db004d1b66f52658548db67760f5e7f4a43f5322b1b00894052e6b01f"} Oct 31 00:52:53.611294 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:53.611209 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:53.611294 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:53.611277 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:53.612624 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:53.612599 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Oct 31 00:52:53.638512 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:53.638464 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" podStartSLOduration=2.603222809 podStartE2EDuration="6.638449284s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.035226475s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:53.638104476 +0000 UTC m=+1386.127379655" watchObservedRunningTime="2026-10-31 00:52:53.638449284 +0000 UTC m=+1386.127724465" Oct 31 00:52:54.616227 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:54.616184 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Oct 31 00:52:55.388594 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:55.388554 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" podUID="09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused" Oct 31 00:52:59.622065 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:59.622039 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:52:59.622584 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:52:59.622559 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Oct 31 00:53:00.388222 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:00.388180 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" podUID="09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused" Oct 31 00:53:00.388404 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:00.388301 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:53:00.508991 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:53:00.508953 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:53:05.388653 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:05.388612 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" podUID="09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused" Oct 31 00:53:09.622764 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:09.622721 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Oct 31 00:53:10.388519 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:10.388475 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" podUID="09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused" Oct 31 00:53:15.388569 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:15.388524 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" podUID="09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused" Oct 31 00:53:17.723178 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:53:17.723142 5939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09422611_0fe3_4cf1_8f0c_bba1b9ee1a24.slice/crio-cdc12c8dc20750b72aed24bddfce553862a8ae2ac95d00b0d4abfe96442706fa\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09422611_0fe3_4cf1_8f0c_bba1b9ee1a24.slice/crio-15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09422611_0fe3_4cf1_8f0c_bba1b9ee1a24.slice/crio-conmon-15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f84625_11a2_4de1_8d23_45555705aedd.slice/crio-886c034c4680d07edd2d9e122dc6cf6ffb7bcf40caf72e22868c4924593a06dc.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:53:17.723723 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:53:17.723157 5939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f84625_11a2_4de1_8d23_45555705aedd.slice/crio-886c034c4680d07edd2d9e122dc6cf6ffb7bcf40caf72e22868c4924593a06dc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09422611_0fe3_4cf1_8f0c_bba1b9ee1a24.slice/crio-conmon-15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09422611_0fe3_4cf1_8f0c_bba1b9ee1a24.slice/crio-15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f84625_11a2_4de1_8d23_45555705aedd.slice/crio-conmon-886c034c4680d07edd2d9e122dc6cf6ffb7bcf40caf72e22868c4924593a06dc.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:53:17.723723 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:53:17.723211 5939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09422611_0fe3_4cf1_8f0c_bba1b9ee1a24.slice/crio-15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f84625_11a2_4de1_8d23_45555705aedd.slice/crio-conmon-886c034c4680d07edd2d9e122dc6cf6ffb7bcf40caf72e22868c4924593a06dc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f84625_11a2_4de1_8d23_45555705aedd.slice/crio-886c034c4680d07edd2d9e122dc6cf6ffb7bcf40caf72e22868c4924593a06dc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09422611_0fe3_4cf1_8f0c_bba1b9ee1a24.slice/crio-conmon-15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:53:17.723723 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:53:17.723139 5939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f84625_11a2_4de1_8d23_45555705aedd.slice/crio-conmon-886c034c4680d07edd2d9e122dc6cf6ffb7bcf40caf72e22868c4924593a06dc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f84625_11a2_4de1_8d23_45555705aedd.slice/crio-886c034c4680d07edd2d9e122dc6cf6ffb7bcf40caf72e22868c4924593a06dc.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:53:17.723935 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:53:17.723802 5939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09422611_0fe3_4cf1_8f0c_bba1b9ee1a24.slice/crio-cdc12c8dc20750b72aed24bddfce553862a8ae2ac95d00b0d4abfe96442706fa\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09422611_0fe3_4cf1_8f0c_bba1b9ee1a24.slice/crio-conmon-15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09422611_0fe3_4cf1_8f0c_bba1b9ee1a24.slice/crio-15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:53:17.724989 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:17.724961 5939 generic.go:356] "Generic (PLEG): container finished" podID="e8f84625-11a2-4de1-8d23-45555705aedd" containerID="886c034c4680d07edd2d9e122dc6cf6ffb7bcf40caf72e22868c4924593a06dc" exitCode=137 Oct 31 00:53:17.725113 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:17.725049 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl" event={"ID":"e8f84625-11a2-4de1-8d23-45555705aedd","Type":"ContainerDied","Data":"886c034c4680d07edd2d9e122dc6cf6ffb7bcf40caf72e22868c4924593a06dc"} Oct 31 00:53:17.725113 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:17.725092 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl" event={"ID":"e8f84625-11a2-4de1-8d23-45555705aedd","Type":"ContainerDied","Data":"4c7cb398ba38c67c73bceeae0c92e4007a903ce88767138d337115c356573124"} Oct 31 00:53:17.725113 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:17.725111 5939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c7cb398ba38c67c73bceeae0c92e4007a903ce88767138d337115c356573124" Oct 31 00:53:17.731520 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:17.731497 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl" Oct 31 00:53:17.818530 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:17.818423 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8f84625-11a2-4de1-8d23-45555705aedd-kube-api-access-9cn4s\" (UniqueName: \"kubernetes.io/projected/e8f84625-11a2-4de1-8d23-45555705aedd-kube-api-access-9cn4s\") pod \"e8f84625-11a2-4de1-8d23-45555705aedd\" (UID: \"e8f84625-11a2-4de1-8d23-45555705aedd\") " Oct 31 00:53:17.820971 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:17.820942 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f84625-11a2-4de1-8d23-45555705aedd-kube-api-access-9cn4s" pod "e8f84625-11a2-4de1-8d23-45555705aedd" (UID: "e8f84625-11a2-4de1-8d23-45555705aedd"). InnerVolumeSpecName "kube-api-access-9cn4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:17.866984 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:17.866957 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:53:17.919123 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:17.919092 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-proxy-tls\" (UniqueName: \"kubernetes.io/secret/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-proxy-tls\") pod \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\" (UID: \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\") " Oct 31 00:53:17.919297 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:17.919262 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81\") pod \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\" (UID: \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\") " Oct 31 00:53:17.919358 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:17.919333 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-kube-api-access-d69dc\" (UniqueName: \"kubernetes.io/projected/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-kube-api-access-d69dc\") pod \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\" (UID: \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\") " Oct 31 00:53:17.919417 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:17.919407 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-kube-rbac-proxy-sar-config\") pod \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\" (UID: \"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24\") " Oct 31 00:53:17.919685 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:17.919645 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cn4s\" (UniqueName: \"kubernetes.io/projected/e8f84625-11a2-4de1-8d23-45555705aedd-kube-api-access-9cn4s\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:17.919866 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:17.919839 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-kube-rbac-proxy-sar-config" pod "09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" (UID: "09422611-0fe3-4cf1-8f0c-bba1b9ee1a24"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:17.921744 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:17.921716 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-proxy-tls" pod "09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" (UID: "09422611-0fe3-4cf1-8f0c-bba1b9ee1a24"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:53:17.922059 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:17.921908 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-kube-api-access-d69dc" pod "09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" (UID: "09422611-0fe3-4cf1-8f0c-bba1b9ee1a24"). InnerVolumeSpecName "kube-api-access-d69dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:17.922828 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:17.922805 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81" pod "09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" (UID: "09422611-0fe3-4cf1-8f0c-bba1b9ee1a24"). InnerVolumeSpecName "pvc-15d6681b-85cb-48db-836d-919ed7bc2c32". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:53:18.020640 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.020603 5939 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-15d6681b-85cb-48db-836d-919ed7bc2c32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" " Oct 31 00:53:18.020640 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.020637 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d69dc\" (UniqueName: \"kubernetes.io/projected/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-kube-api-access-d69dc\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:18.020845 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.020651 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:18.020845 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.020678 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:18.031457 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.031436 5939 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-15d6681b-85cb-48db-836d-919ed7bc2c32" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81") on node "ip-10-0-143-213.us-east-2.compute.internal" Oct 31 00:53:18.121377 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.121347 5939 reconciler_common.go:299] "Volume detached for volume \"pvc-15d6681b-85cb-48db-836d-919ed7bc2c32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-068304bd85b0b8d81\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"csi-04d74cf946b08ebd8bafd32b63259630b25bc98904a0b26b1c7728f22278636c\"" Oct 31 00:53:18.732361 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.732328 5939 generic.go:356] "Generic (PLEG): container finished" podID="09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" containerID="15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9" exitCode=137 Oct 31 00:53:18.732806 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.732402 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" Oct 31 00:53:18.732806 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.732414 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" event={"ID":"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24","Type":"ContainerDied","Data":"15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9"} Oct 31 00:53:18.732806 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.732482 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb" event={"ID":"09422611-0fe3-4cf1-8f0c-bba1b9ee1a24","Type":"ContainerDied","Data":"cdc12c8dc20750b72aed24bddfce553862a8ae2ac95d00b0d4abfe96442706fa"} Oct 31 00:53:18.732806 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.732521 5939 scope.go:122] "RemoveContainer" containerID="0ce3d648cfdb3481c999ad5aae787602ca6423c1ce17301220211c97cc73e4e2" Oct 31 00:53:18.732806 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.732659 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl" Oct 31 00:53:18.741889 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.741870 5939 scope.go:122] "RemoveContainer" containerID="15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9" Oct 31 00:53:18.749514 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.749495 5939 scope.go:122] "RemoveContainer" containerID="0ce3d648cfdb3481c999ad5aae787602ca6423c1ce17301220211c97cc73e4e2" Oct 31 00:53:18.749784 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:53:18.749766 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce3d648cfdb3481c999ad5aae787602ca6423c1ce17301220211c97cc73e4e2\": container with ID starting with 0ce3d648cfdb3481c999ad5aae787602ca6423c1ce17301220211c97cc73e4e2 not found: ID does not exist" containerID="0ce3d648cfdb3481c999ad5aae787602ca6423c1ce17301220211c97cc73e4e2" Oct 31 00:53:18.749847 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.749793 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce3d648cfdb3481c999ad5aae787602ca6423c1ce17301220211c97cc73e4e2"} err="failed to get container status \"0ce3d648cfdb3481c999ad5aae787602ca6423c1ce17301220211c97cc73e4e2\": rpc error: code = NotFound desc = could not find container \"0ce3d648cfdb3481c999ad5aae787602ca6423c1ce17301220211c97cc73e4e2\": container with ID starting with 0ce3d648cfdb3481c999ad5aae787602ca6423c1ce17301220211c97cc73e4e2 not found: ID does not exist" Oct 31 00:53:18.749847 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.749808 5939 scope.go:122] "RemoveContainer" containerID="15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9" Oct 31 00:53:18.750042 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:53:18.750026 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9\": container with ID starting with 15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9 not found: ID does not exist" containerID="15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9" Oct 31 00:53:18.750086 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.750048 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9"} err="failed to get container status \"15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9\": rpc error: code = NotFound desc = could not find container \"15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9\": container with ID starting with 15b4afbb3b483e7691888064914288c368a736715a3bbe6be12369a02aed26f9 not found: ID does not exist" Oct 31 00:53:18.760578 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.760558 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl"] Oct 31 00:53:18.766603 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.766582 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-sink-bdd864ff9-lf5dl"] Oct 31 00:53:18.782444 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.782424 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb"] Oct 31 00:53:18.788262 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:18.788242 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/proxy-routing-0e21ef4e-predictor-69ffcb56f8-hlhtb"] Oct 31 00:53:19.622809 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:19.622769 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Oct 31 00:53:19.652343 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:19.652314 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" path="/var/lib/kubelet/pods/09422611-0fe3-4cf1-8f0c-bba1b9ee1a24/volumes" Oct 31 00:53:19.652703 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:19.652687 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8f84625-11a2-4de1-8d23-45555705aedd" path="/var/lib/kubelet/pods/e8f84625-11a2-4de1-8d23-45555705aedd/volumes" Oct 31 00:53:29.622495 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:29.622456 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Oct 31 00:53:30.509309 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:53:30.509274 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:53:39.623428 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:39.623385 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Oct 31 00:53:49.622799 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:49.622761 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Oct 31 00:53:59.623116 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:53:59.623034 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Oct 31 00:54:00.507869 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:54:00.507829 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:54:00.648094 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:00.648054 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Oct 31 00:54:10.648606 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:10.648577 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:54:18.098767 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:18.098734 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-predictor-6fdbb7885f-jvx7g_1b17ead2-b3ae-485f-bd37-0e838656ca9e/kserve-container/0.log" Oct 31 00:54:20.124346 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:20.124315 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-predictor-6fdbb7885f-jvx7g_1b17ead2-b3ae-485f-bd37-0e838656ca9e/kserve-container/0.log" Oct 31 00:54:22.149896 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:22.149869 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-predictor-6fdbb7885f-jvx7g_1b17ead2-b3ae-485f-bd37-0e838656ca9e/kserve-container/0.log" Oct 31 00:54:24.175403 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:24.175368 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-predictor-6fdbb7885f-jvx7g_1b17ead2-b3ae-485f-bd37-0e838656ca9e/kserve-container/0.log" Oct 31 00:54:26.206280 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:26.206253 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-predictor-6fdbb7885f-jvx7g_1b17ead2-b3ae-485f-bd37-0e838656ca9e/kserve-container/0.log" Oct 31 00:54:28.231842 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:28.231812 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-predictor-6fdbb7885f-jvx7g_1b17ead2-b3ae-485f-bd37-0e838656ca9e/kserve-container/0.log" Oct 31 00:54:30.257750 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:30.257721 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-predictor-6fdbb7885f-jvx7g_1b17ead2-b3ae-485f-bd37-0e838656ca9e/kserve-container/0.log" Oct 31 00:54:30.508626 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:54:30.508546 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:54:32.284085 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:32.284056 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-predictor-6fdbb7885f-jvx7g_1b17ead2-b3ae-485f-bd37-0e838656ca9e/kserve-container/0.log" Oct 31 00:54:34.309988 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:34.309959 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-predictor-6fdbb7885f-jvx7g_1b17ead2-b3ae-485f-bd37-0e838656ca9e/kserve-container/0.log" Oct 31 00:54:36.335268 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:36.335229 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-predictor-6fdbb7885f-jvx7g_1b17ead2-b3ae-485f-bd37-0e838656ca9e/kserve-container/0.log" Oct 31 00:54:38.360150 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:38.360120 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-predictor-6fdbb7885f-jvx7g_1b17ead2-b3ae-485f-bd37-0e838656ca9e/kserve-container/0.log" Oct 31 00:54:40.384637 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:40.384612 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-predictor-6fdbb7885f-jvx7g_1b17ead2-b3ae-485f-bd37-0e838656ca9e/kserve-container/0.log" Oct 31 00:54:42.409464 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:42.409430 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-predictor-6fdbb7885f-jvx7g_1b17ead2-b3ae-485f-bd37-0e838656ca9e/kserve-container/0.log" Oct 31 00:54:44.434315 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:44.434283 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-predictor-6fdbb7885f-jvx7g_1b17ead2-b3ae-485f-bd37-0e838656ca9e/kserve-container/0.log" Oct 31 00:54:46.459684 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:46.459635 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-predictor-6fdbb7885f-jvx7g_1b17ead2-b3ae-485f-bd37-0e838656ca9e/kserve-container/0.log" Oct 31 00:54:47.824493 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:47.824461 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:54:47.827101 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:47.827084 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:54:48.486466 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:48.486434 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-predictor-6fdbb7885f-jvx7g_1b17ead2-b3ae-485f-bd37-0e838656ca9e/kserve-container/0.log" Oct 31 00:54:48.782493 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:48.782411 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g"] Oct 31 00:54:48.782763 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:48.782713 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kserve-container" containerID="cri-o://7b14998db004d1b66f52658548db67760f5e7f4a43f5322b1b00894052e6b01f" gracePeriod=30 Oct 31 00:54:48.782913 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:48.782812 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kube-rbac-proxy" containerID="cri-o://78d7637f6da9cb47b4f7060d45d18b0b13bfa77f7e8460e5a6c6db201d828479" gracePeriod=30 Oct 31 00:54:49.120244 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:49.120206 5939 generic.go:356] "Generic (PLEG): container finished" podID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerID="78d7637f6da9cb47b4f7060d45d18b0b13bfa77f7e8460e5a6c6db201d828479" exitCode=0 Oct 31 00:54:49.120626 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:49.120273 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" event={"ID":"1b17ead2-b3ae-485f-bd37-0e838656ca9e","Type":"ContainerDied","Data":"78d7637f6da9cb47b4f7060d45d18b0b13bfa77f7e8460e5a6c6db201d828479"} Oct 31 00:54:49.616913 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:49.616870 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused" Oct 31 00:54:50.648104 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:50.648061 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Oct 31 00:54:53.622461 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:53.622430 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:54:53.729007 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:53.728928 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b17ead2-b3ae-485f-bd37-0e838656ca9e-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b17ead2-b3ae-485f-bd37-0e838656ca9e-proxy-tls\") pod \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " Oct 31 00:54:53.729007 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:53.728972 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kube-rbac-proxy-sar-config\") pod \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " Oct 31 00:54:53.729239 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:53.729035 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kserve-provision-location\") pod \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " Oct 31 00:54:53.729239 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:53.729078 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b17ead2-b3ae-485f-bd37-0e838656ca9e-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b17ead2-b3ae-485f-bd37-0e838656ca9e-cabundle-cert\") pod \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " Oct 31 00:54:53.729239 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:53.729106 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kube-api-access-khlxw\" (UniqueName: \"kubernetes.io/projected/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kube-api-access-khlxw\") pod \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\" (UID: \"1b17ead2-b3ae-485f-bd37-0e838656ca9e\") " Oct 31 00:54:53.729402 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:53.729349 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kserve-provision-location" pod "1b17ead2-b3ae-485f-bd37-0e838656ca9e" (UID: "1b17ead2-b3ae-485f-bd37-0e838656ca9e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:53.729467 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:53.729403 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kube-rbac-proxy-sar-config" pod "1b17ead2-b3ae-485f-bd37-0e838656ca9e" (UID: "1b17ead2-b3ae-485f-bd37-0e838656ca9e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:53.729520 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:53.729483 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:53.729520 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:53.729503 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:53.729819 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:53.729783 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b17ead2-b3ae-485f-bd37-0e838656ca9e-cabundle-cert" pod "1b17ead2-b3ae-485f-bd37-0e838656ca9e" (UID: "1b17ead2-b3ae-485f-bd37-0e838656ca9e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:53.731162 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:53.731131 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b17ead2-b3ae-485f-bd37-0e838656ca9e-proxy-tls" pod "1b17ead2-b3ae-485f-bd37-0e838656ca9e" (UID: "1b17ead2-b3ae-485f-bd37-0e838656ca9e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:53.731427 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:53.731406 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kube-api-access-khlxw" pod "1b17ead2-b3ae-485f-bd37-0e838656ca9e" (UID: "1b17ead2-b3ae-485f-bd37-0e838656ca9e"). InnerVolumeSpecName "kube-api-access-khlxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:53.830053 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:53.830015 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b17ead2-b3ae-485f-bd37-0e838656ca9e-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:53.830053 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:53.830047 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b17ead2-b3ae-485f-bd37-0e838656ca9e-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:53.830254 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:53.830063 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khlxw\" (UniqueName: \"kubernetes.io/projected/1b17ead2-b3ae-485f-bd37-0e838656ca9e-kube-api-access-khlxw\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:54.143948 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:54.143909 5939 generic.go:356] "Generic (PLEG): container finished" podID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerID="7b14998db004d1b66f52658548db67760f5e7f4a43f5322b1b00894052e6b01f" exitCode=0 Oct 31 00:54:54.144135 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:54.143994 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" Oct 31 00:54:54.144135 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:54.143997 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" event={"ID":"1b17ead2-b3ae-485f-bd37-0e838656ca9e","Type":"ContainerDied","Data":"7b14998db004d1b66f52658548db67760f5e7f4a43f5322b1b00894052e6b01f"} Oct 31 00:54:54.144135 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:54.144080 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g" event={"ID":"1b17ead2-b3ae-485f-bd37-0e838656ca9e","Type":"ContainerDied","Data":"17547b2d64fed839299880422fec319d2613b242da75c7bd7c0d30f1c9fbd342"} Oct 31 00:54:54.144135 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:54.144116 5939 scope.go:122] "RemoveContainer" containerID="78d7637f6da9cb47b4f7060d45d18b0b13bfa77f7e8460e5a6c6db201d828479" Oct 31 00:54:54.152721 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:54.152701 5939 scope.go:122] "RemoveContainer" containerID="7b14998db004d1b66f52658548db67760f5e7f4a43f5322b1b00894052e6b01f" Oct 31 00:54:54.159929 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:54.159913 5939 scope.go:122] "RemoveContainer" containerID="ae3b22272efb33fb54e033351aa6d6cd10d02c6233753c553ff0de30672e43d2" Oct 31 00:54:54.167591 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:54.167522 5939 scope.go:122] "RemoveContainer" containerID="78d7637f6da9cb47b4f7060d45d18b0b13bfa77f7e8460e5a6c6db201d828479" Oct 31 00:54:54.168007 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:54:54.167980 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78d7637f6da9cb47b4f7060d45d18b0b13bfa77f7e8460e5a6c6db201d828479\": container with ID starting with 78d7637f6da9cb47b4f7060d45d18b0b13bfa77f7e8460e5a6c6db201d828479 not found: ID does not exist" containerID="78d7637f6da9cb47b4f7060d45d18b0b13bfa77f7e8460e5a6c6db201d828479" Oct 31 00:54:54.168098 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:54.168019 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d7637f6da9cb47b4f7060d45d18b0b13bfa77f7e8460e5a6c6db201d828479"} err="failed to get container status \"78d7637f6da9cb47b4f7060d45d18b0b13bfa77f7e8460e5a6c6db201d828479\": rpc error: code = NotFound desc = could not find container \"78d7637f6da9cb47b4f7060d45d18b0b13bfa77f7e8460e5a6c6db201d828479\": container with ID starting with 78d7637f6da9cb47b4f7060d45d18b0b13bfa77f7e8460e5a6c6db201d828479 not found: ID does not exist" Oct 31 00:54:54.168098 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:54.168045 5939 scope.go:122] "RemoveContainer" containerID="7b14998db004d1b66f52658548db67760f5e7f4a43f5322b1b00894052e6b01f" Oct 31 00:54:54.168413 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:54:54.168386 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b14998db004d1b66f52658548db67760f5e7f4a43f5322b1b00894052e6b01f\": container with ID starting with 7b14998db004d1b66f52658548db67760f5e7f4a43f5322b1b00894052e6b01f not found: ID does not exist" containerID="7b14998db004d1b66f52658548db67760f5e7f4a43f5322b1b00894052e6b01f" Oct 31 00:54:54.168961 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:54.168419 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b14998db004d1b66f52658548db67760f5e7f4a43f5322b1b00894052e6b01f"} err="failed to get container status \"7b14998db004d1b66f52658548db67760f5e7f4a43f5322b1b00894052e6b01f\": rpc error: code = NotFound desc = could not find container \"7b14998db004d1b66f52658548db67760f5e7f4a43f5322b1b00894052e6b01f\": container with ID starting with 7b14998db004d1b66f52658548db67760f5e7f4a43f5322b1b00894052e6b01f not found: ID does not exist" Oct 31 00:54:54.168961 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:54.168439 5939 scope.go:122] "RemoveContainer" containerID="ae3b22272efb33fb54e033351aa6d6cd10d02c6233753c553ff0de30672e43d2" Oct 31 00:54:54.168961 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:54:54.168748 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae3b22272efb33fb54e033351aa6d6cd10d02c6233753c553ff0de30672e43d2\": container with ID starting with ae3b22272efb33fb54e033351aa6d6cd10d02c6233753c553ff0de30672e43d2 not found: ID does not exist" containerID="ae3b22272efb33fb54e033351aa6d6cd10d02c6233753c553ff0de30672e43d2" Oct 31 00:54:54.168961 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:54.168773 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae3b22272efb33fb54e033351aa6d6cd10d02c6233753c553ff0de30672e43d2"} err="failed to get container status \"ae3b22272efb33fb54e033351aa6d6cd10d02c6233753c553ff0de30672e43d2\": rpc error: code = NotFound desc = could not find container \"ae3b22272efb33fb54e033351aa6d6cd10d02c6233753c553ff0de30672e43d2\": container with ID starting with ae3b22272efb33fb54e033351aa6d6cd10d02c6233753c553ff0de30672e43d2 not found: ID does not exist" Oct 31 00:54:54.169613 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:54.169595 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g"] Oct 31 00:54:54.175771 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:54.175749 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-predictor-6fdbb7885f-jvx7g"] Oct 31 00:54:55.651925 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:55.651895 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" path="/var/lib/kubelet/pods/1b17ead2-b3ae-485f-bd37-0e838656ca9e/volumes" Oct 31 00:54:56.275592 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.275559 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t"] Oct 31 00:54:56.275917 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.275904 5939 cpu_manager.go:414] "containerMap: removing container" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kserve-container" Oct 31 00:54:56.275967 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.275919 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kserve-container" Oct 31 00:54:56.275967 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.275931 5939 cpu_manager.go:414] "containerMap: removing container" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kube-rbac-proxy" Oct 31 00:54:56.275967 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.275936 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kube-rbac-proxy" Oct 31 00:54:56.275967 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.275943 5939 cpu_manager.go:414] "containerMap: removing container" podUID="09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" containerName="kube-rbac-proxy" Oct 31 00:54:56.275967 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.275950 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" containerName="kube-rbac-proxy" Oct 31 00:54:56.275967 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.275959 5939 cpu_manager.go:414] "containerMap: removing container" podUID="e8f84625-11a2-4de1-8d23-45555705aedd" containerName="kserve-container" Oct 31 00:54:56.275967 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.275963 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8f84625-11a2-4de1-8d23-45555705aedd" containerName="kserve-container" Oct 31 00:54:56.276180 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.275971 5939 cpu_manager.go:414] "containerMap: removing container" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="storage-initializer" Oct 31 00:54:56.276180 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.275975 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="storage-initializer" Oct 31 00:54:56.276180 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.275981 5939 cpu_manager.go:414] "containerMap: removing container" podUID="09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" containerName="kserve-container" Oct 31 00:54:56.276180 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.275985 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" containerName="kserve-container" Oct 31 00:54:56.276180 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.276052 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kserve-container" Oct 31 00:54:56.276180 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.276059 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b17ead2-b3ae-485f-bd37-0e838656ca9e" containerName="kube-rbac-proxy" Oct 31 00:54:56.276180 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.276065 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" containerName="kube-rbac-proxy" Oct 31 00:54:56.276180 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.276071 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8f84625-11a2-4de1-8d23-45555705aedd" containerName="kserve-container" Oct 31 00:54:56.276180 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.276077 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="09422611-0fe3-4cf1-8f0c-bba1b9ee1a24" containerName="kserve-container" Oct 31 00:54:56.280803 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.280783 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:54:56.286806 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.286773 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:54:56.287662 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.287644 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 00:54:56.287776 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.287644 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\"" Oct 31 00:54:56.287776 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.287652 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 00:54:56.287776 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.287734 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 00:54:56.287776 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.287744 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"sklearn-v2-mlserver-predictor-serving-cert\"" Oct 31 00:54:56.287988 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.287796 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 00:54:56.293102 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.293081 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t"] Oct 31 00:54:56.351042 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.351013 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bzbn\" (UniqueName: \"kubernetes.io/projected/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kube-api-access-6bzbn\") pod \"sklearn-v2-mlserver-predictor-7745c95c5c-76h2t\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:54:56.351042 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.351045 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7745c95c5c-76h2t\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:54:56.351221 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.351072 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7745c95c5c-76h2t\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:54:56.351221 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.351184 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7745c95c5c-76h2t\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:54:56.351221 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.351205 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7745c95c5c-76h2t\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:54:56.451966 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.451938 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bzbn\" (UniqueName: \"kubernetes.io/projected/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kube-api-access-6bzbn\") pod \"sklearn-v2-mlserver-predictor-7745c95c5c-76h2t\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:54:56.451966 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.451975 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7745c95c5c-76h2t\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:54:56.452186 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.452014 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7745c95c5c-76h2t\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:54:56.452186 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.452054 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7745c95c5c-76h2t\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:54:56.452186 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.452080 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7745c95c5c-76h2t\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:54:56.452469 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.452448 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7745c95c5c-76h2t\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:54:56.452774 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.452753 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7745c95c5c-76h2t\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:54:56.452846 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.452759 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7745c95c5c-76h2t\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:54:56.454633 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.454606 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7745c95c5c-76h2t\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:54:56.466578 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.466551 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bzbn\" (UniqueName: \"kubernetes.io/projected/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kube-api-access-6bzbn\") pod \"sklearn-v2-mlserver-predictor-7745c95c5c-76h2t\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:54:56.598271 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.598186 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:54:56.879509 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:56.879478 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t"] Oct 31 00:54:56.882548 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:54:56.882522 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0dbb1d_44e1_43ed_a472_7786c382f1a1.slice/crio-231ac4cdba6db43541afbfb4d803bb9d8bf341ec2347a78f47235f92dea885ef WatchSource:0}: Error finding container 231ac4cdba6db43541afbfb4d803bb9d8bf341ec2347a78f47235f92dea885ef: Status 404 returned error can't find the container with id 231ac4cdba6db43541afbfb4d803bb9d8bf341ec2347a78f47235f92dea885ef Oct 31 00:54:57.159906 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:57.159802 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" event={"ID":"ba0dbb1d-44e1-43ed-a472-7786c382f1a1","Type":"ContainerStarted","Data":"4c0f7acf3ef5291f53e0365183deb4511b3385d5bfb14843897b889ff7a6be2a"} Oct 31 00:54:57.159906 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:54:57.159883 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" event={"ID":"ba0dbb1d-44e1-43ed-a472-7786c382f1a1","Type":"ContainerStarted","Data":"231ac4cdba6db43541afbfb4d803bb9d8bf341ec2347a78f47235f92dea885ef"} Oct 31 00:55:00.509293 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:55:00.509247 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:55:01.176727 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:01.176694 5939 generic.go:356] "Generic (PLEG): container finished" podID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" containerID="4c0f7acf3ef5291f53e0365183deb4511b3385d5bfb14843897b889ff7a6be2a" exitCode=0 Oct 31 00:55:01.176892 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:01.176733 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" event={"ID":"ba0dbb1d-44e1-43ed-a472-7786c382f1a1","Type":"ContainerDied","Data":"4c0f7acf3ef5291f53e0365183deb4511b3385d5bfb14843897b889ff7a6be2a"} Oct 31 00:55:30.512955 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:55:30.512871 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:55:46.925771 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:46.925723 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4"] Oct 31 00:55:46.960059 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:46.960017 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4"] Oct 31 00:55:46.960234 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:46.960152 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:46.966647 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:46.966623 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\"" Oct 31 00:55:46.966803 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:46.966720 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\"" Oct 31 00:55:46.967477 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:46.967457 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-logger-kube-rbac-proxy-sar-config\"" Oct 31 00:55:46.967563 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:46.967494 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-logger-predictor-serving-cert\"" Oct 31 00:55:46.967604 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:46.967564 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\"" Oct 31 00:55:46.967604 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:46.967574 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\"" Oct 31 00:55:46.967703 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:46.967624 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:55:47.029754 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.029661 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e79f087b-3814-436b-9822-4d3cbf956e58-cabundle-cert\") pod \"isvc-logger-predictor-7b4f4cc5cc-92pb4\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:47.029926 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.029809 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e79f087b-3814-436b-9822-4d3cbf956e58-kserve-provision-location\") pod \"isvc-logger-predictor-7b4f4cc5cc-92pb4\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:47.029926 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.029866 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e79f087b-3814-436b-9822-4d3cbf956e58-proxy-tls\") pod \"isvc-logger-predictor-7b4f4cc5cc-92pb4\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:47.030119 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.029960 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e79f087b-3814-436b-9822-4d3cbf956e58-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7b4f4cc5cc-92pb4\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:47.030119 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.030030 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsslk\" (UniqueName: \"kubernetes.io/projected/e79f087b-3814-436b-9822-4d3cbf956e58-kube-api-access-lsslk\") pod \"isvc-logger-predictor-7b4f4cc5cc-92pb4\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:47.130454 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.130419 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e79f087b-3814-436b-9822-4d3cbf956e58-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7b4f4cc5cc-92pb4\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:47.130652 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.130473 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsslk\" (UniqueName: \"kubernetes.io/projected/e79f087b-3814-436b-9822-4d3cbf956e58-kube-api-access-lsslk\") pod \"isvc-logger-predictor-7b4f4cc5cc-92pb4\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:47.130652 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.130539 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e79f087b-3814-436b-9822-4d3cbf956e58-cabundle-cert\") pod \"isvc-logger-predictor-7b4f4cc5cc-92pb4\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:47.130652 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.130574 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e79f087b-3814-436b-9822-4d3cbf956e58-kserve-provision-location\") pod \"isvc-logger-predictor-7b4f4cc5cc-92pb4\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:47.131027 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.130722 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e79f087b-3814-436b-9822-4d3cbf956e58-proxy-tls\") pod \"isvc-logger-predictor-7b4f4cc5cc-92pb4\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:47.131203 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.131182 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e79f087b-3814-436b-9822-4d3cbf956e58-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7b4f4cc5cc-92pb4\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:47.131570 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.131491 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e79f087b-3814-436b-9822-4d3cbf956e58-kserve-provision-location\") pod \"isvc-logger-predictor-7b4f4cc5cc-92pb4\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:47.131829 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.131800 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e79f087b-3814-436b-9822-4d3cbf956e58-cabundle-cert\") pod \"isvc-logger-predictor-7b4f4cc5cc-92pb4\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:47.137198 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.137176 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e79f087b-3814-436b-9822-4d3cbf956e58-proxy-tls\") pod \"isvc-logger-predictor-7b4f4cc5cc-92pb4\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:47.146902 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.146874 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsslk\" (UniqueName: \"kubernetes.io/projected/e79f087b-3814-436b-9822-4d3cbf956e58-kube-api-access-lsslk\") pod \"isvc-logger-predictor-7b4f4cc5cc-92pb4\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:47.347254 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.347178 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:47.658414 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.658384 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4"] Oct 31 00:55:47.662376 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:55:47.662345 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79f087b_3814_436b_9822_4d3cbf956e58.slice/crio-b1bf36fb166446d97375e77e4d3388e60b76095cc07ee6cbb94e0392c262db90 WatchSource:0}: Error finding container b1bf36fb166446d97375e77e4d3388e60b76095cc07ee6cbb94e0392c262db90: Status 404 returned error can't find the container with id b1bf36fb166446d97375e77e4d3388e60b76095cc07ee6cbb94e0392c262db90 Oct 31 00:55:47.667900 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:47.667880 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\"" Oct 31 00:55:48.444075 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:48.444031 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" event={"ID":"e79f087b-3814-436b-9822-4d3cbf956e58","Type":"ContainerStarted","Data":"f3280ea1c55a24e43cbd650888c10340b075152188504685772413f875da29b2"} Oct 31 00:55:48.444574 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:48.444115 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" event={"ID":"e79f087b-3814-436b-9822-4d3cbf956e58","Type":"ContainerStarted","Data":"b1bf36fb166446d97375e77e4d3388e60b76095cc07ee6cbb94e0392c262db90"} Oct 31 00:55:52.466351 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:52.466308 5939 generic.go:356] "Generic (PLEG): container finished" podID="e79f087b-3814-436b-9822-4d3cbf956e58" containerID="f3280ea1c55a24e43cbd650888c10340b075152188504685772413f875da29b2" exitCode=0 Oct 31 00:55:52.466895 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:52.466349 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" event={"ID":"e79f087b-3814-436b-9822-4d3cbf956e58","Type":"ContainerDied","Data":"f3280ea1c55a24e43cbd650888c10340b075152188504685772413f875da29b2"} Oct 31 00:55:53.474884 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:53.474851 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" event={"ID":"e79f087b-3814-436b-9822-4d3cbf956e58","Type":"ContainerStarted","Data":"4b98d92c954ee4df05a19d133bc4eb01f8da966f41c5eb6f1f8bf71204d62204"} Oct 31 00:55:53.475334 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:53.474923 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" event={"ID":"e79f087b-3814-436b-9822-4d3cbf956e58","Type":"ContainerStarted","Data":"b3a6b53b8601fb5a350b08a8cfc3858ad820ef1805622122a1b0547f424c0f11"} Oct 31 00:55:53.475334 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:53.474957 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" event={"ID":"e79f087b-3814-436b-9822-4d3cbf956e58","Type":"ContainerStarted","Data":"ab964838ab008f5913fc1eecc857e9fbd97f854a15e311606da3a0f16a8764d7"} Oct 31 00:55:53.475334 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:53.475181 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:53.507772 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:53.507719 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podStartSLOduration=3.3903962659999998 podStartE2EDuration="7.507704673s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.117308407s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:53.505526396 +0000 UTC m=+1565.994801575" watchObservedRunningTime="2026-10-31 00:55:53.507704673 +0000 UTC m=+1565.996979842" Oct 31 00:55:54.480728 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:54.480689 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:54.481196 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:54.480760 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:55:54.482118 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:54.482086 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Oct 31 00:55:54.482908 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:54.482873 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:55:55.485944 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:55.485892 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Oct 31 00:55:55.486603 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:55:55.486575 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:00.491637 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:00.491605 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:56:00.492480 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:00.492438 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Oct 31 00:56:00.492869 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:00.492840 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:00.513659 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:56:00.513627 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:56:10.492580 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:10.492541 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Oct 31 00:56:10.493190 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:10.493150 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:20.492563 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:20.492519 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Oct 31 00:56:20.493067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:20.492983 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:30.492317 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:30.492263 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Oct 31 00:56:30.492940 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:30.492785 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:30.513648 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:56:30.513607 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:56:40.492930 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:40.492892 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Oct 31 00:56:40.493531 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:40.493440 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:50.492316 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:50.492261 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Oct 31 00:56:50.493129 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:50.493095 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:59.808989 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:59.808956 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" event={"ID":"ba0dbb1d-44e1-43ed-a472-7786c382f1a1","Type":"ContainerStarted","Data":"ba4eae701a69058bbd2eafe44d65da4711de7809e88343a281d887245d6094cc"} Oct 31 00:56:59.809480 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:59.809019 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" event={"ID":"ba0dbb1d-44e1-43ed-a472-7786c382f1a1","Type":"ContainerStarted","Data":"06fdcc2e5b485e97014d62c5c6ec219a12b71a19f47cb065cc5ef07a4c2cfd91"} Oct 31 00:56:59.809480 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:59.809059 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:56:59.809480 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:59.809202 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:56:59.839894 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:56:59.839850 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" podStartSLOduration=1.760169491 podStartE2EDuration="2m3.839838649s" totalImagesPullingTime="1m58.064778246s" totalInitContainerRuntime="4.014890912s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:59.838257001 +0000 UTC m=+1632.327532179" watchObservedRunningTime="2026-10-31 00:56:59.839838649 +0000 UTC m=+1632.329113819" Oct 31 00:57:00.493077 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:00.493035 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Oct 31 00:57:00.493590 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:00.493567 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:57:00.510359 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:57:00.510325 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:57:05.820811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:05.820781 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:57:09.647270 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:09.647221 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Oct 31 00:57:09.647752 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:09.647726 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:57:19.647883 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:19.647837 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Oct 31 00:57:19.648482 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:19.648255 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:57:29.652194 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:29.652165 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:57:29.652602 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:29.652249 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:57:30.509893 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:57:30.509847 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:57:35.915615 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:35.915572 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" podUID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400" Oct 31 00:57:42.850202 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:42.850172 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4"] Oct 31 00:57:42.850632 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:42.850406 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" containerID="cri-o://ab964838ab008f5913fc1eecc857e9fbd97f854a15e311606da3a0f16a8764d7" gracePeriod=30 Oct 31 00:57:42.850632 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:42.850449 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" containerID="cri-o://4b98d92c954ee4df05a19d133bc4eb01f8da966f41c5eb6f1f8bf71204d62204" gracePeriod=30 Oct 31 00:57:42.850632 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:42.850471 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kube-rbac-proxy" containerID="cri-o://b3a6b53b8601fb5a350b08a8cfc3858ad820ef1805622122a1b0547f424c0f11" gracePeriod=30 Oct 31 00:57:43.005469 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:43.005439 5939 generic.go:356] "Generic (PLEG): container finished" podID="e79f087b-3814-436b-9822-4d3cbf956e58" containerID="b3a6b53b8601fb5a350b08a8cfc3858ad820ef1805622122a1b0547f424c0f11" exitCode=0 Oct 31 00:57:43.005595 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:43.005513 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" event={"ID":"e79f087b-3814-436b-9822-4d3cbf956e58","Type":"ContainerDied","Data":"b3a6b53b8601fb5a350b08a8cfc3858ad820ef1805622122a1b0547f424c0f11"} Oct 31 00:57:45.486143 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:45.486101 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused" Oct 31 00:57:45.823372 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:45.823297 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:57:47.887717 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:47.887690 5939 scope.go:122] "RemoveContainer" containerID="886c034c4680d07edd2d9e122dc6cf6ffb7bcf40caf72e22868c4924593a06dc" Oct 31 00:57:48.029935 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:48.029907 5939 generic.go:356] "Generic (PLEG): container finished" podID="e79f087b-3814-436b-9822-4d3cbf956e58" containerID="ab964838ab008f5913fc1eecc857e9fbd97f854a15e311606da3a0f16a8764d7" exitCode=0 Oct 31 00:57:48.030054 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:48.029978 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" event={"ID":"e79f087b-3814-436b-9822-4d3cbf956e58","Type":"ContainerDied","Data":"ab964838ab008f5913fc1eecc857e9fbd97f854a15e311606da3a0f16a8764d7"} Oct 31 00:57:49.647419 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:49.647373 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Oct 31 00:57:49.647856 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:49.647719 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:57:50.485892 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:50.485850 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused" Oct 31 00:57:50.789325 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:50.789251 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t"] Oct 31 00:57:50.789733 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:50.789503 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" podUID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" containerName="kserve-container" containerID="cri-o://06fdcc2e5b485e97014d62c5c6ec219a12b71a19f47cb065cc5ef07a4c2cfd91" gracePeriod=30 Oct 31 00:57:50.789733 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:50.789566 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" podUID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" containerName="kube-rbac-proxy" containerID="cri-o://ba4eae701a69058bbd2eafe44d65da4711de7809e88343a281d887245d6094cc" gracePeriod=30 Oct 31 00:57:50.814495 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:50.814464 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" podUID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused" Oct 31 00:57:51.044339 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:51.044249 5939 generic.go:356] "Generic (PLEG): container finished" podID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" containerID="ba4eae701a69058bbd2eafe44d65da4711de7809e88343a281d887245d6094cc" exitCode=0 Oct 31 00:57:51.044339 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:51.044321 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" event={"ID":"ba0dbb1d-44e1-43ed-a472-7786c382f1a1","Type":"ContainerDied","Data":"ba4eae701a69058bbd2eafe44d65da4711de7809e88343a281d887245d6094cc"} Oct 31 00:57:55.486316 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:55.486272 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused" Oct 31 00:57:55.486784 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:55.486402 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:57:55.814769 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:55.814661 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" podUID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused" Oct 31 00:57:55.821501 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:55.821476 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" podUID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.44:8080/v2/models/sklearn-v2-mlserver/ready\": dial tcp 10.134.0.44:8080: connect: connection refused" Oct 31 00:57:57.624594 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:57.624568 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:57:57.736263 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:57.736180 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-cabundle-cert\") pod \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " Oct 31 00:57:57.736263 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:57.736222 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-proxy-tls\") pod \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " Oct 31 00:57:57.736263 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:57.736246 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kube-rbac-proxy-sar-config\") pod \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " Oct 31 00:57:57.736536 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:57.736331 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kube-api-access-6bzbn\" (UniqueName: \"kubernetes.io/projected/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kube-api-access-6bzbn\") pod \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " Oct 31 00:57:57.736536 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:57.736387 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kserve-provision-location\") pod \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\" (UID: \"ba0dbb1d-44e1-43ed-a472-7786c382f1a1\") " Oct 31 00:57:57.736648 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:57.736598 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-cabundle-cert" pod "ba0dbb1d-44e1-43ed-a472-7786c382f1a1" (UID: "ba0dbb1d-44e1-43ed-a472-7786c382f1a1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:57.736739 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:57.736628 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kube-rbac-proxy-sar-config" pod "ba0dbb1d-44e1-43ed-a472-7786c382f1a1" (UID: "ba0dbb1d-44e1-43ed-a472-7786c382f1a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:57.736792 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:57.736757 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kserve-provision-location" pod "ba0dbb1d-44e1-43ed-a472-7786c382f1a1" (UID: "ba0dbb1d-44e1-43ed-a472-7786c382f1a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:57.738373 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:57.738348 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-proxy-tls" pod "ba0dbb1d-44e1-43ed-a472-7786c382f1a1" (UID: "ba0dbb1d-44e1-43ed-a472-7786c382f1a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:57.738461 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:57.738365 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kube-api-access-6bzbn" pod "ba0dbb1d-44e1-43ed-a472-7786c382f1a1" (UID: "ba0dbb1d-44e1-43ed-a472-7786c382f1a1"). InnerVolumeSpecName "kube-api-access-6bzbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:57.837526 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:57.837497 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:57.837526 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:57.837521 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bzbn\" (UniqueName: \"kubernetes.io/projected/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kube-api-access-6bzbn\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:57.837526 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:57.837532 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:57.837732 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:57.837541 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:57.837732 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:57.837550 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba0dbb1d-44e1-43ed-a472-7786c382f1a1-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:58.076413 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:58.076332 5939 generic.go:356] "Generic (PLEG): container finished" podID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" containerID="06fdcc2e5b485e97014d62c5c6ec219a12b71a19f47cb065cc5ef07a4c2cfd91" exitCode=0 Oct 31 00:57:58.076413 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:58.076385 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" Oct 31 00:57:58.076576 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:58.076419 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" event={"ID":"ba0dbb1d-44e1-43ed-a472-7786c382f1a1","Type":"ContainerDied","Data":"06fdcc2e5b485e97014d62c5c6ec219a12b71a19f47cb065cc5ef07a4c2cfd91"} Oct 31 00:57:58.076576 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:58.076477 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t" event={"ID":"ba0dbb1d-44e1-43ed-a472-7786c382f1a1","Type":"ContainerDied","Data":"231ac4cdba6db43541afbfb4d803bb9d8bf341ec2347a78f47235f92dea885ef"} Oct 31 00:57:58.076576 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:58.076504 5939 scope.go:122] "RemoveContainer" containerID="ba4eae701a69058bbd2eafe44d65da4711de7809e88343a281d887245d6094cc" Oct 31 00:57:58.085294 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:58.085276 5939 scope.go:122] "RemoveContainer" containerID="06fdcc2e5b485e97014d62c5c6ec219a12b71a19f47cb065cc5ef07a4c2cfd91" Oct 31 00:57:58.092441 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:58.092424 5939 scope.go:122] "RemoveContainer" containerID="4c0f7acf3ef5291f53e0365183deb4511b3385d5bfb14843897b889ff7a6be2a" Oct 31 00:57:58.102047 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:58.101926 5939 scope.go:122] "RemoveContainer" containerID="ba4eae701a69058bbd2eafe44d65da4711de7809e88343a281d887245d6094cc" Oct 31 00:57:58.102286 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:57:58.102262 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba4eae701a69058bbd2eafe44d65da4711de7809e88343a281d887245d6094cc\": container with ID starting with ba4eae701a69058bbd2eafe44d65da4711de7809e88343a281d887245d6094cc not found: ID does not exist" containerID="ba4eae701a69058bbd2eafe44d65da4711de7809e88343a281d887245d6094cc" Oct 31 00:57:58.102372 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:58.102295 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba4eae701a69058bbd2eafe44d65da4711de7809e88343a281d887245d6094cc"} err="failed to get container status \"ba4eae701a69058bbd2eafe44d65da4711de7809e88343a281d887245d6094cc\": rpc error: code = NotFound desc = could not find container \"ba4eae701a69058bbd2eafe44d65da4711de7809e88343a281d887245d6094cc\": container with ID starting with ba4eae701a69058bbd2eafe44d65da4711de7809e88343a281d887245d6094cc not found: ID does not exist" Oct 31 00:57:58.102372 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:58.102316 5939 scope.go:122] "RemoveContainer" containerID="06fdcc2e5b485e97014d62c5c6ec219a12b71a19f47cb065cc5ef07a4c2cfd91" Oct 31 00:57:58.102911 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:57:58.102888 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06fdcc2e5b485e97014d62c5c6ec219a12b71a19f47cb065cc5ef07a4c2cfd91\": container with ID starting with 06fdcc2e5b485e97014d62c5c6ec219a12b71a19f47cb065cc5ef07a4c2cfd91 not found: ID does not exist" containerID="06fdcc2e5b485e97014d62c5c6ec219a12b71a19f47cb065cc5ef07a4c2cfd91" Oct 31 00:57:58.103013 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:58.102915 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06fdcc2e5b485e97014d62c5c6ec219a12b71a19f47cb065cc5ef07a4c2cfd91"} err="failed to get container status \"06fdcc2e5b485e97014d62c5c6ec219a12b71a19f47cb065cc5ef07a4c2cfd91\": rpc error: code = NotFound desc = could not find container \"06fdcc2e5b485e97014d62c5c6ec219a12b71a19f47cb065cc5ef07a4c2cfd91\": container with ID starting with 06fdcc2e5b485e97014d62c5c6ec219a12b71a19f47cb065cc5ef07a4c2cfd91 not found: ID does not exist" Oct 31 00:57:58.103013 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:58.102930 5939 scope.go:122] "RemoveContainer" containerID="4c0f7acf3ef5291f53e0365183deb4511b3385d5bfb14843897b889ff7a6be2a" Oct 31 00:57:58.103216 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:57:58.103197 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c0f7acf3ef5291f53e0365183deb4511b3385d5bfb14843897b889ff7a6be2a\": container with ID starting with 4c0f7acf3ef5291f53e0365183deb4511b3385d5bfb14843897b889ff7a6be2a not found: ID does not exist" containerID="4c0f7acf3ef5291f53e0365183deb4511b3385d5bfb14843897b889ff7a6be2a" Oct 31 00:57:58.103270 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:58.103225 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c0f7acf3ef5291f53e0365183deb4511b3385d5bfb14843897b889ff7a6be2a"} err="failed to get container status \"4c0f7acf3ef5291f53e0365183deb4511b3385d5bfb14843897b889ff7a6be2a\": rpc error: code = NotFound desc = could not find container \"4c0f7acf3ef5291f53e0365183deb4511b3385d5bfb14843897b889ff7a6be2a\": container with ID starting with 4c0f7acf3ef5291f53e0365183deb4511b3385d5bfb14843897b889ff7a6be2a not found: ID does not exist" Oct 31 00:57:58.103341 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:58.103324 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t"] Oct 31 00:57:58.109596 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:58.109574 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/sklearn-v2-mlserver-predictor-7745c95c5c-76h2t"] Oct 31 00:57:59.647975 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:59.647925 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Oct 31 00:57:59.648511 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:59.648388 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:57:59.651828 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:57:59.651802 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" path="/var/lib/kubelet/pods/ba0dbb1d-44e1-43ed-a472-7786c382f1a1/volumes" Oct 31 00:58:00.397038 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.397010 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl"] Oct 31 00:58:00.397346 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.397334 5939 cpu_manager.go:414] "containerMap: removing container" podUID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" containerName="storage-initializer" Oct 31 00:58:00.397400 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.397347 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" containerName="storage-initializer" Oct 31 00:58:00.397400 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.397369 5939 cpu_manager.go:414] "containerMap: removing container" podUID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" containerName="kube-rbac-proxy" Oct 31 00:58:00.397400 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.397376 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" containerName="kube-rbac-proxy" Oct 31 00:58:00.397400 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.397389 5939 cpu_manager.go:414] "containerMap: removing container" podUID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" containerName="kserve-container" Oct 31 00:58:00.397400 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.397394 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" containerName="kserve-container" Oct 31 00:58:00.397558 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.397475 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" containerName="kube-rbac-proxy" Oct 31 00:58:00.397558 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.397483 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0dbb1d-44e1-43ed-a472-7786c382f1a1" containerName="kserve-container" Oct 31 00:58:00.400464 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.400445 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:00.409963 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.409940 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 00:58:00.409963 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.409950 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:58:00.410517 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.410497 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 00:58:00.410631 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.410619 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 00:58:00.410707 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.410635 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\"" Oct 31 00:58:00.410707 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.410697 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-runtime-predictor-serving-cert\"" Oct 31 00:58:00.411552 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.411540 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 00:58:00.414011 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.413956 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl"] Oct 31 00:58:00.461387 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.461366 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srtzn\" (UniqueName: \"kubernetes.io/projected/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kube-api-access-srtzn\") pod \"isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:00.461506 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.461408 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:00.461506 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.461435 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4c74a20-a65a-470e-85a2-245c67a1b6ba-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:00.461583 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.461540 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4c74a20-a65a-470e-85a2-245c67a1b6ba-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:00.461583 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.461569 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:00.485946 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.485921 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused" Oct 31 00:58:00.509099 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:58:00.509068 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:58:00.562872 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.562844 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4c74a20-a65a-470e-85a2-245c67a1b6ba-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:00.563022 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.562884 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4c74a20-a65a-470e-85a2-245c67a1b6ba-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:00.563022 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.562906 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:00.563022 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.562957 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srtzn\" (UniqueName: \"kubernetes.io/projected/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kube-api-access-srtzn\") pod \"isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:00.563022 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.562983 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:00.563423 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.563401 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:00.563663 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.563643 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4c74a20-a65a-470e-85a2-245c67a1b6ba-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:00.563742 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.563696 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:00.565450 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.565424 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4c74a20-a65a-470e-85a2-245c67a1b6ba-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:00.577546 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.577523 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srtzn\" (UniqueName: \"kubernetes.io/projected/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kube-api-access-srtzn\") pod \"isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:00.714719 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.714624 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:00.993146 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:00.993075 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl"] Oct 31 00:58:00.996345 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:58:00.996316 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c74a20_a65a_470e_85a2_245c67a1b6ba.slice/crio-48d9accc969707a4001da4d1e2d467fb142c26b716c8a2201e067f57ddefa5b6 WatchSource:0}: Error finding container 48d9accc969707a4001da4d1e2d467fb142c26b716c8a2201e067f57ddefa5b6: Status 404 returned error can't find the container with id 48d9accc969707a4001da4d1e2d467fb142c26b716c8a2201e067f57ddefa5b6 Oct 31 00:58:01.092357 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:01.092319 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" event={"ID":"e4c74a20-a65a-470e-85a2-245c67a1b6ba","Type":"ContainerStarted","Data":"d117063e840bce630db24318520d0f8c983579c52556344a24fd344d8b58aca6"} Oct 31 00:58:01.092523 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:01.092386 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" event={"ID":"e4c74a20-a65a-470e-85a2-245c67a1b6ba","Type":"ContainerStarted","Data":"48d9accc969707a4001da4d1e2d467fb142c26b716c8a2201e067f57ddefa5b6"} Oct 31 00:58:05.486221 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:05.486175 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused" Oct 31 00:58:09.130704 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:09.130344 5939 generic.go:356] "Generic (PLEG): container finished" podID="e4c74a20-a65a-470e-85a2-245c67a1b6ba" containerID="d117063e840bce630db24318520d0f8c983579c52556344a24fd344d8b58aca6" exitCode=0 Oct 31 00:58:09.130704 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:09.130397 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" event={"ID":"e4c74a20-a65a-470e-85a2-245c67a1b6ba","Type":"ContainerDied","Data":"d117063e840bce630db24318520d0f8c983579c52556344a24fd344d8b58aca6"} Oct 31 00:58:09.647434 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:09.647382 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Oct 31 00:58:09.647714 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:09.647695 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:58:09.653311 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:09.653287 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:58:09.653422 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:09.653400 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:58:10.137382 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:10.137347 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" event={"ID":"e4c74a20-a65a-470e-85a2-245c67a1b6ba","Type":"ContainerStarted","Data":"7c1158ab495b26031a3496a8e20417ec8b7ea252dd2889eca1e59545ead8a82a"} Oct 31 00:58:10.137815 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:10.137448 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" event={"ID":"e4c74a20-a65a-470e-85a2-245c67a1b6ba","Type":"ContainerStarted","Data":"d8c5672d4a1fd6592c3d3ce0323ec34b1724d3b54b96c57ed0f673823369fd05"} Oct 31 00:58:10.137815 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:10.137753 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:10.137936 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:10.137916 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:10.139047 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:10.139025 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" podUID="e4c74a20-a65a-470e-85a2-245c67a1b6ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused" Oct 31 00:58:10.164442 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:10.164398 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" podStartSLOduration=2.272052787 podStartE2EDuration="10.164380958s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.892328171s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:10.163785578 +0000 UTC m=+1702.653060751" watchObservedRunningTime="2026-10-31 00:58:10.164380958 +0000 UTC m=+1702.653660097" Oct 31 00:58:10.485947 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:10.485852 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused" Oct 31 00:58:11.142029 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:11.141987 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" podUID="e4c74a20-a65a-470e-85a2-245c67a1b6ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused" Oct 31 00:58:13.004811 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.004787 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:58:13.153277 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.153244 5939 generic.go:356] "Generic (PLEG): container finished" podID="e79f087b-3814-436b-9822-4d3cbf956e58" containerID="4b98d92c954ee4df05a19d133bc4eb01f8da966f41c5eb6f1f8bf71204d62204" exitCode=137 Oct 31 00:58:13.153423 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.153300 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" event={"ID":"e79f087b-3814-436b-9822-4d3cbf956e58","Type":"ContainerDied","Data":"4b98d92c954ee4df05a19d133bc4eb01f8da966f41c5eb6f1f8bf71204d62204"} Oct 31 00:58:13.153423 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.153325 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" Oct 31 00:58:13.153423 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.153358 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4" event={"ID":"e79f087b-3814-436b-9822-4d3cbf956e58","Type":"ContainerDied","Data":"b1bf36fb166446d97375e77e4d3388e60b76095cc07ee6cbb94e0392c262db90"} Oct 31 00:58:13.153423 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.153391 5939 scope.go:122] "RemoveContainer" containerID="4b98d92c954ee4df05a19d133bc4eb01f8da966f41c5eb6f1f8bf71204d62204" Oct 31 00:58:13.161456 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.161427 5939 scope.go:122] "RemoveContainer" containerID="b3a6b53b8601fb5a350b08a8cfc3858ad820ef1805622122a1b0547f424c0f11" Oct 31 00:58:13.168878 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.168861 5939 scope.go:122] "RemoveContainer" containerID="ab964838ab008f5913fc1eecc857e9fbd97f854a15e311606da3a0f16a8764d7" Oct 31 00:58:13.175948 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.175931 5939 scope.go:122] "RemoveContainer" containerID="f3280ea1c55a24e43cbd650888c10340b075152188504685772413f875da29b2" Oct 31 00:58:13.179068 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.179041 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79f087b-3814-436b-9822-4d3cbf956e58-kube-api-access-lsslk\" (UniqueName: \"kubernetes.io/projected/e79f087b-3814-436b-9822-4d3cbf956e58-kube-api-access-lsslk\") pod \"e79f087b-3814-436b-9822-4d3cbf956e58\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " Oct 31 00:58:13.179176 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.179089 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e79f087b-3814-436b-9822-4d3cbf956e58-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e79f087b-3814-436b-9822-4d3cbf956e58-proxy-tls\") pod \"e79f087b-3814-436b-9822-4d3cbf956e58\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " Oct 31 00:58:13.179244 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.179215 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e79f087b-3814-436b-9822-4d3cbf956e58-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e79f087b-3814-436b-9822-4d3cbf956e58-cabundle-cert\") pod \"e79f087b-3814-436b-9822-4d3cbf956e58\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " Oct 31 00:58:13.179305 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.179245 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e79f087b-3814-436b-9822-4d3cbf956e58-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e79f087b-3814-436b-9822-4d3cbf956e58-kube-rbac-proxy-sar-config\") pod \"e79f087b-3814-436b-9822-4d3cbf956e58\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " Oct 31 00:58:13.179305 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.179277 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79f087b-3814-436b-9822-4d3cbf956e58-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e79f087b-3814-436b-9822-4d3cbf956e58-kserve-provision-location\") pod \"e79f087b-3814-436b-9822-4d3cbf956e58\" (UID: \"e79f087b-3814-436b-9822-4d3cbf956e58\") " Oct 31 00:58:13.179606 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.179580 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79f087b-3814-436b-9822-4d3cbf956e58-kube-rbac-proxy-sar-config" pod "e79f087b-3814-436b-9822-4d3cbf956e58" (UID: "e79f087b-3814-436b-9822-4d3cbf956e58"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:13.179711 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.179611 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79f087b-3814-436b-9822-4d3cbf956e58-kserve-provision-location" pod "e79f087b-3814-436b-9822-4d3cbf956e58" (UID: "e79f087b-3814-436b-9822-4d3cbf956e58"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:13.179711 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.179615 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79f087b-3814-436b-9822-4d3cbf956e58-cabundle-cert" pod "e79f087b-3814-436b-9822-4d3cbf956e58" (UID: "e79f087b-3814-436b-9822-4d3cbf956e58"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:13.181283 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.181249 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79f087b-3814-436b-9822-4d3cbf956e58-kube-api-access-lsslk" pod "e79f087b-3814-436b-9822-4d3cbf956e58" (UID: "e79f087b-3814-436b-9822-4d3cbf956e58"). InnerVolumeSpecName "kube-api-access-lsslk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:13.181513 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.181496 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e79f087b-3814-436b-9822-4d3cbf956e58-proxy-tls" pod "e79f087b-3814-436b-9822-4d3cbf956e58" (UID: "e79f087b-3814-436b-9822-4d3cbf956e58"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:13.184179 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.184144 5939 scope.go:122] "RemoveContainer" containerID="4b98d92c954ee4df05a19d133bc4eb01f8da966f41c5eb6f1f8bf71204d62204" Oct 31 00:58:13.184467 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:58:13.184440 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b98d92c954ee4df05a19d133bc4eb01f8da966f41c5eb6f1f8bf71204d62204\": container with ID starting with 4b98d92c954ee4df05a19d133bc4eb01f8da966f41c5eb6f1f8bf71204d62204 not found: ID does not exist" containerID="4b98d92c954ee4df05a19d133bc4eb01f8da966f41c5eb6f1f8bf71204d62204" Oct 31 00:58:13.184542 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.184476 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b98d92c954ee4df05a19d133bc4eb01f8da966f41c5eb6f1f8bf71204d62204"} err="failed to get container status \"4b98d92c954ee4df05a19d133bc4eb01f8da966f41c5eb6f1f8bf71204d62204\": rpc error: code = NotFound desc = could not find container \"4b98d92c954ee4df05a19d133bc4eb01f8da966f41c5eb6f1f8bf71204d62204\": container with ID starting with 4b98d92c954ee4df05a19d133bc4eb01f8da966f41c5eb6f1f8bf71204d62204 not found: ID does not exist" Oct 31 00:58:13.184542 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.184492 5939 scope.go:122] "RemoveContainer" containerID="b3a6b53b8601fb5a350b08a8cfc3858ad820ef1805622122a1b0547f424c0f11" Oct 31 00:58:13.184746 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:58:13.184722 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3a6b53b8601fb5a350b08a8cfc3858ad820ef1805622122a1b0547f424c0f11\": container with ID starting with b3a6b53b8601fb5a350b08a8cfc3858ad820ef1805622122a1b0547f424c0f11 not found: ID does not exist" containerID="b3a6b53b8601fb5a350b08a8cfc3858ad820ef1805622122a1b0547f424c0f11" Oct 31 00:58:13.184842 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.184750 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3a6b53b8601fb5a350b08a8cfc3858ad820ef1805622122a1b0547f424c0f11"} err="failed to get container status \"b3a6b53b8601fb5a350b08a8cfc3858ad820ef1805622122a1b0547f424c0f11\": rpc error: code = NotFound desc = could not find container \"b3a6b53b8601fb5a350b08a8cfc3858ad820ef1805622122a1b0547f424c0f11\": container with ID starting with b3a6b53b8601fb5a350b08a8cfc3858ad820ef1805622122a1b0547f424c0f11 not found: ID does not exist" Oct 31 00:58:13.184842 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.184766 5939 scope.go:122] "RemoveContainer" containerID="ab964838ab008f5913fc1eecc857e9fbd97f854a15e311606da3a0f16a8764d7" Oct 31 00:58:13.185034 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:58:13.185014 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab964838ab008f5913fc1eecc857e9fbd97f854a15e311606da3a0f16a8764d7\": container with ID starting with ab964838ab008f5913fc1eecc857e9fbd97f854a15e311606da3a0f16a8764d7 not found: ID does not exist" containerID="ab964838ab008f5913fc1eecc857e9fbd97f854a15e311606da3a0f16a8764d7" Oct 31 00:58:13.185109 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.185036 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab964838ab008f5913fc1eecc857e9fbd97f854a15e311606da3a0f16a8764d7"} err="failed to get container status \"ab964838ab008f5913fc1eecc857e9fbd97f854a15e311606da3a0f16a8764d7\": rpc error: code = NotFound desc = could not find container \"ab964838ab008f5913fc1eecc857e9fbd97f854a15e311606da3a0f16a8764d7\": container with ID starting with ab964838ab008f5913fc1eecc857e9fbd97f854a15e311606da3a0f16a8764d7 not found: ID does not exist" Oct 31 00:58:13.185109 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.185050 5939 scope.go:122] "RemoveContainer" containerID="f3280ea1c55a24e43cbd650888c10340b075152188504685772413f875da29b2" Oct 31 00:58:13.185290 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:58:13.185272 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3280ea1c55a24e43cbd650888c10340b075152188504685772413f875da29b2\": container with ID starting with f3280ea1c55a24e43cbd650888c10340b075152188504685772413f875da29b2 not found: ID does not exist" containerID="f3280ea1c55a24e43cbd650888c10340b075152188504685772413f875da29b2" Oct 31 00:58:13.185377 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.185296 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3280ea1c55a24e43cbd650888c10340b075152188504685772413f875da29b2"} err="failed to get container status \"f3280ea1c55a24e43cbd650888c10340b075152188504685772413f875da29b2\": rpc error: code = NotFound desc = could not find container \"f3280ea1c55a24e43cbd650888c10340b075152188504685772413f875da29b2\": container with ID starting with f3280ea1c55a24e43cbd650888c10340b075152188504685772413f875da29b2 not found: ID does not exist" Oct 31 00:58:13.280048 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.280021 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e79f087b-3814-436b-9822-4d3cbf956e58-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:13.280048 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.280045 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e79f087b-3814-436b-9822-4d3cbf956e58-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:13.280048 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.280057 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e79f087b-3814-436b-9822-4d3cbf956e58-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:13.280362 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.280066 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e79f087b-3814-436b-9822-4d3cbf956e58-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:13.280362 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.280075 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsslk\" (UniqueName: \"kubernetes.io/projected/e79f087b-3814-436b-9822-4d3cbf956e58-kube-api-access-lsslk\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:13.480486 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.480451 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4"] Oct 31 00:58:13.487332 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.487306 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-logger-predictor-7b4f4cc5cc-92pb4"] Oct 31 00:58:13.651525 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:13.651492 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" path="/var/lib/kubelet/pods/e79f087b-3814-436b-9822-4d3cbf956e58/volumes" Oct 31 00:58:16.146875 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:16.146847 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:16.147393 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:16.147367 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" podUID="e4c74a20-a65a-470e-85a2-245c67a1b6ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused" Oct 31 00:58:26.148812 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:26.148783 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:30.508892 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:58:30.508813 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:58:33.834421 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:33.834390 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl"] Oct 31 00:58:33.834847 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:33.834627 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" podUID="e4c74a20-a65a-470e-85a2-245c67a1b6ba" containerName="kserve-container" containerID="cri-o://d8c5672d4a1fd6592c3d3ce0323ec34b1724d3b54b96c57ed0f673823369fd05" gracePeriod=30 Oct 31 00:58:33.834847 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:33.834685 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" podUID="e4c74a20-a65a-470e-85a2-245c67a1b6ba" containerName="kube-rbac-proxy" containerID="cri-o://7c1158ab495b26031a3496a8e20417ec8b7ea252dd2889eca1e59545ead8a82a" gracePeriod=30 Oct 31 00:58:34.257615 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:34.257577 5939 generic.go:356] "Generic (PLEG): container finished" podID="e4c74a20-a65a-470e-85a2-245c67a1b6ba" containerID="7c1158ab495b26031a3496a8e20417ec8b7ea252dd2889eca1e59545ead8a82a" exitCode=0 Oct 31 00:58:34.257790 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:34.257648 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" event={"ID":"e4c74a20-a65a-470e-85a2-245c67a1b6ba","Type":"ContainerDied","Data":"7c1158ab495b26031a3496a8e20417ec8b7ea252dd2889eca1e59545ead8a82a"} Oct 31 00:58:34.979054 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:34.979029 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:35.068016 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.067985 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c74a20-a65a-470e-85a2-245c67a1b6ba-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4c74a20-a65a-470e-85a2-245c67a1b6ba-cabundle-cert\") pod \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " Oct 31 00:58:35.068184 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.068056 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kube-api-access-srtzn\" (UniqueName: \"kubernetes.io/projected/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kube-api-access-srtzn\") pod \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " Oct 31 00:58:35.068184 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.068088 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kserve-provision-location\") pod \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " Oct 31 00:58:35.068184 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.068128 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kube-rbac-proxy-sar-config\") pod \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " Oct 31 00:58:35.068358 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.068196 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4c74a20-a65a-470e-85a2-245c67a1b6ba-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4c74a20-a65a-470e-85a2-245c67a1b6ba-proxy-tls\") pod \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\" (UID: \"e4c74a20-a65a-470e-85a2-245c67a1b6ba\") " Oct 31 00:58:35.068468 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.068399 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c74a20-a65a-470e-85a2-245c67a1b6ba-cabundle-cert" pod "e4c74a20-a65a-470e-85a2-245c67a1b6ba" (UID: "e4c74a20-a65a-470e-85a2-245c67a1b6ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:35.068611 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.068563 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kube-rbac-proxy-sar-config" pod "e4c74a20-a65a-470e-85a2-245c67a1b6ba" (UID: "e4c74a20-a65a-470e-85a2-245c67a1b6ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:35.070229 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.070207 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c74a20-a65a-470e-85a2-245c67a1b6ba-proxy-tls" pod "e4c74a20-a65a-470e-85a2-245c67a1b6ba" (UID: "e4c74a20-a65a-470e-85a2-245c67a1b6ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:35.070321 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.070244 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kube-api-access-srtzn" pod "e4c74a20-a65a-470e-85a2-245c67a1b6ba" (UID: "e4c74a20-a65a-470e-85a2-245c67a1b6ba"). InnerVolumeSpecName "kube-api-access-srtzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:35.095067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.095012 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kserve-provision-location" pod "e4c74a20-a65a-470e-85a2-245c67a1b6ba" (UID: "e4c74a20-a65a-470e-85a2-245c67a1b6ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:35.169640 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.169614 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4c74a20-a65a-470e-85a2-245c67a1b6ba-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:35.169640 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.169637 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4c74a20-a65a-470e-85a2-245c67a1b6ba-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:35.169823 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.169648 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srtzn\" (UniqueName: \"kubernetes.io/projected/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kube-api-access-srtzn\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:35.169823 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.169658 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:35.169823 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.169684 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4c74a20-a65a-470e-85a2-245c67a1b6ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:35.263716 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.263683 5939 generic.go:356] "Generic (PLEG): container finished" podID="e4c74a20-a65a-470e-85a2-245c67a1b6ba" containerID="d8c5672d4a1fd6592c3d3ce0323ec34b1724d3b54b96c57ed0f673823369fd05" exitCode=0 Oct 31 00:58:35.263874 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.263741 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" Oct 31 00:58:35.263874 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.263769 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" event={"ID":"e4c74a20-a65a-470e-85a2-245c67a1b6ba","Type":"ContainerDied","Data":"d8c5672d4a1fd6592c3d3ce0323ec34b1724d3b54b96c57ed0f673823369fd05"} Oct 31 00:58:35.263874 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.263835 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl" event={"ID":"e4c74a20-a65a-470e-85a2-245c67a1b6ba","Type":"ContainerDied","Data":"48d9accc969707a4001da4d1e2d467fb142c26b716c8a2201e067f57ddefa5b6"} Oct 31 00:58:35.263874 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.263863 5939 scope.go:122] "RemoveContainer" containerID="7c1158ab495b26031a3496a8e20417ec8b7ea252dd2889eca1e59545ead8a82a" Oct 31 00:58:35.272790 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.272772 5939 scope.go:122] "RemoveContainer" containerID="d8c5672d4a1fd6592c3d3ce0323ec34b1724d3b54b96c57ed0f673823369fd05" Oct 31 00:58:35.280191 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.280176 5939 scope.go:122] "RemoveContainer" containerID="d117063e840bce630db24318520d0f8c983579c52556344a24fd344d8b58aca6" Oct 31 00:58:35.286982 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.286967 5939 scope.go:122] "RemoveContainer" containerID="7c1158ab495b26031a3496a8e20417ec8b7ea252dd2889eca1e59545ead8a82a" Oct 31 00:58:35.287211 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:58:35.287195 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c1158ab495b26031a3496a8e20417ec8b7ea252dd2889eca1e59545ead8a82a\": container with ID starting with 7c1158ab495b26031a3496a8e20417ec8b7ea252dd2889eca1e59545ead8a82a not found: ID does not exist" containerID="7c1158ab495b26031a3496a8e20417ec8b7ea252dd2889eca1e59545ead8a82a" Oct 31 00:58:35.287263 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.287222 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c1158ab495b26031a3496a8e20417ec8b7ea252dd2889eca1e59545ead8a82a"} err="failed to get container status \"7c1158ab495b26031a3496a8e20417ec8b7ea252dd2889eca1e59545ead8a82a\": rpc error: code = NotFound desc = could not find container \"7c1158ab495b26031a3496a8e20417ec8b7ea252dd2889eca1e59545ead8a82a\": container with ID starting with 7c1158ab495b26031a3496a8e20417ec8b7ea252dd2889eca1e59545ead8a82a not found: ID does not exist" Oct 31 00:58:35.287263 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.287238 5939 scope.go:122] "RemoveContainer" containerID="d8c5672d4a1fd6592c3d3ce0323ec34b1724d3b54b96c57ed0f673823369fd05" Oct 31 00:58:35.287445 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:58:35.287425 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c5672d4a1fd6592c3d3ce0323ec34b1724d3b54b96c57ed0f673823369fd05\": container with ID starting with d8c5672d4a1fd6592c3d3ce0323ec34b1724d3b54b96c57ed0f673823369fd05 not found: ID does not exist" containerID="d8c5672d4a1fd6592c3d3ce0323ec34b1724d3b54b96c57ed0f673823369fd05" Oct 31 00:58:35.287510 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.287455 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c5672d4a1fd6592c3d3ce0323ec34b1724d3b54b96c57ed0f673823369fd05"} err="failed to get container status \"d8c5672d4a1fd6592c3d3ce0323ec34b1724d3b54b96c57ed0f673823369fd05\": rpc error: code = NotFound desc = could not find container \"d8c5672d4a1fd6592c3d3ce0323ec34b1724d3b54b96c57ed0f673823369fd05\": container with ID starting with d8c5672d4a1fd6592c3d3ce0323ec34b1724d3b54b96c57ed0f673823369fd05 not found: ID does not exist" Oct 31 00:58:35.287510 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.287477 5939 scope.go:122] "RemoveContainer" containerID="d117063e840bce630db24318520d0f8c983579c52556344a24fd344d8b58aca6" Oct 31 00:58:35.287701 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:58:35.287686 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d117063e840bce630db24318520d0f8c983579c52556344a24fd344d8b58aca6\": container with ID starting with d117063e840bce630db24318520d0f8c983579c52556344a24fd344d8b58aca6 not found: ID does not exist" containerID="d117063e840bce630db24318520d0f8c983579c52556344a24fd344d8b58aca6" Oct 31 00:58:35.287755 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.287704 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d117063e840bce630db24318520d0f8c983579c52556344a24fd344d8b58aca6"} err="failed to get container status \"d117063e840bce630db24318520d0f8c983579c52556344a24fd344d8b58aca6\": rpc error: code = NotFound desc = could not find container \"d117063e840bce630db24318520d0f8c983579c52556344a24fd344d8b58aca6\": container with ID starting with d117063e840bce630db24318520d0f8c983579c52556344a24fd344d8b58aca6 not found: ID does not exist" Oct 31 00:58:35.290410 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.290390 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl"] Oct 31 00:58:35.296834 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.296815 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-runtime-predictor-65b64dbd69-wtzvl"] Oct 31 00:58:35.651486 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:35.651456 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c74a20-a65a-470e-85a2-245c67a1b6ba" path="/var/lib/kubelet/pods/e4c74a20-a65a-470e-85a2-245c67a1b6ba/volumes" Oct 31 00:58:37.243494 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.243456 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz"] Oct 31 00:58:37.243937 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.243918 5939 cpu_manager.go:414] "containerMap: removing container" podUID="e4c74a20-a65a-470e-85a2-245c67a1b6ba" containerName="storage-initializer" Oct 31 00:58:37.243937 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.243933 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c74a20-a65a-470e-85a2-245c67a1b6ba" containerName="storage-initializer" Oct 31 00:58:37.244015 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.243943 5939 cpu_manager.go:414] "containerMap: removing container" podUID="e4c74a20-a65a-470e-85a2-245c67a1b6ba" containerName="kube-rbac-proxy" Oct 31 00:58:37.244015 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.243948 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c74a20-a65a-470e-85a2-245c67a1b6ba" containerName="kube-rbac-proxy" Oct 31 00:58:37.244015 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.243957 5939 cpu_manager.go:414] "containerMap: removing container" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" Oct 31 00:58:37.244015 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.243962 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" Oct 31 00:58:37.244015 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.243982 5939 cpu_manager.go:414] "containerMap: removing container" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="storage-initializer" Oct 31 00:58:37.244015 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.243986 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="storage-initializer" Oct 31 00:58:37.244015 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.243992 5939 cpu_manager.go:414] "containerMap: removing container" podUID="e4c74a20-a65a-470e-85a2-245c67a1b6ba" containerName="kserve-container" Oct 31 00:58:37.244015 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.243996 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c74a20-a65a-470e-85a2-245c67a1b6ba" containerName="kserve-container" Oct 31 00:58:37.244015 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.244009 5939 cpu_manager.go:414] "containerMap: removing container" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" Oct 31 00:58:37.244015 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.244013 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" Oct 31 00:58:37.244015 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.244019 5939 cpu_manager.go:414] "containerMap: removing container" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kube-rbac-proxy" Oct 31 00:58:37.244500 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.244023 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kube-rbac-proxy" Oct 31 00:58:37.244500 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.244078 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c74a20-a65a-470e-85a2-245c67a1b6ba" containerName="kserve-container" Oct 31 00:58:37.244500 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.244095 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kserve-container" Oct 31 00:58:37.244500 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.244101 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="kube-rbac-proxy" Oct 31 00:58:37.244500 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.244110 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c74a20-a65a-470e-85a2-245c67a1b6ba" containerName="kube-rbac-proxy" Oct 31 00:58:37.244500 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.244115 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79f087b-3814-436b-9822-4d3cbf956e58" containerName="agent" Oct 31 00:58:37.248720 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.248702 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:37.254558 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.254534 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:58:37.254782 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.254758 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 00:58:37.255815 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.255795 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\"" Oct 31 00:58:37.255980 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.255946 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 00:58:37.255980 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.255951 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 00:58:37.255980 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.255975 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 00:58:37.256150 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.255979 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\"" Oct 31 00:58:37.261345 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.261325 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz"] Oct 31 00:58:37.287164 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.287143 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:37.287274 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.287185 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30d818ac-ab96-45c6-a666-6dbd36d2ada2-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:37.287274 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.287261 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:37.287346 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.287286 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30d818ac-ab96-45c6-a666-6dbd36d2ada2-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:37.287346 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.287305 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npw7c\" (UniqueName: \"kubernetes.io/projected/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kube-api-access-npw7c\") pod \"isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:37.388363 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.388331 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30d818ac-ab96-45c6-a666-6dbd36d2ada2-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:37.388363 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.388365 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npw7c\" (UniqueName: \"kubernetes.io/projected/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kube-api-access-npw7c\") pod \"isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:37.388587 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.388395 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:37.388587 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.388422 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30d818ac-ab96-45c6-a666-6dbd36d2ada2-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:37.388587 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.388465 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:37.388951 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.388926 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:37.389083 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.389065 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:37.389155 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.389132 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30d818ac-ab96-45c6-a666-6dbd36d2ada2-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:37.390846 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.390823 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30d818ac-ab96-45c6-a666-6dbd36d2ada2-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:37.402864 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.402844 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npw7c\" (UniqueName: \"kubernetes.io/projected/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kube-api-access-npw7c\") pod \"isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:37.571085 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.571002 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:37.852281 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:37.852203 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz"] Oct 31 00:58:37.855494 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:58:37.855468 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d818ac_ab96_45c6_a666_6dbd36d2ada2.slice/crio-eb9f2b6ba245567a122855a7a989b4515a0d95d1d34c86cf146f51a30cceced4 WatchSource:0}: Error finding container eb9f2b6ba245567a122855a7a989b4515a0d95d1d34c86cf146f51a30cceced4: Status 404 returned error can't find the container with id eb9f2b6ba245567a122855a7a989b4515a0d95d1d34c86cf146f51a30cceced4 Oct 31 00:58:38.279733 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:38.279701 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" event={"ID":"30d818ac-ab96-45c6-a666-6dbd36d2ada2","Type":"ContainerStarted","Data":"a016de0dc8eac997e65d6b599ad75073b20178f1de817d1bb02e7ed0e427b2c6"} Oct 31 00:58:38.280106 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:38.279762 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" event={"ID":"30d818ac-ab96-45c6-a666-6dbd36d2ada2","Type":"ContainerStarted","Data":"eb9f2b6ba245567a122855a7a989b4515a0d95d1d34c86cf146f51a30cceced4"} Oct 31 00:58:42.298704 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:42.298655 5939 generic.go:356] "Generic (PLEG): container finished" podID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" containerID="a016de0dc8eac997e65d6b599ad75073b20178f1de817d1bb02e7ed0e427b2c6" exitCode=0 Oct 31 00:58:42.299086 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:42.298722 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" event={"ID":"30d818ac-ab96-45c6-a666-6dbd36d2ada2","Type":"ContainerDied","Data":"a016de0dc8eac997e65d6b599ad75073b20178f1de817d1bb02e7ed0e427b2c6"} Oct 31 00:58:43.305549 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:43.305510 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" event={"ID":"30d818ac-ab96-45c6-a666-6dbd36d2ada2","Type":"ContainerStarted","Data":"93987857d09b71357811921831ca8da2cacaafe59e92e46c2b4c9b6c46c9ca9b"} Oct 31 00:58:43.305956 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:43.305572 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" event={"ID":"30d818ac-ab96-45c6-a666-6dbd36d2ada2","Type":"ContainerStarted","Data":"d662dd6d82ae3440881896b7e45039736ae9d77d92dcfb656ce1d8c4dba39914"} Oct 31 00:58:43.305956 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:43.305904 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:43.305956 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:43.305948 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:58:43.335848 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:43.335807 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" podStartSLOduration=2.535316221 podStartE2EDuration="6.335795988s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.800479767s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:43.333799909 +0000 UTC m=+1735.823075092" watchObservedRunningTime="2026-10-31 00:58:43.335795988 +0000 UTC m=+1735.825071161" Oct 31 00:58:47.905946 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:47.905914 5939 scope.go:122] "RemoveContainer" containerID="d2e740f7b6ade9f8a3cb5977ac87575ef4425b8a484249b60ddd64ca3aaff55f" Oct 31 00:58:47.913329 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:47.913307 5939 scope.go:122] "RemoveContainer" containerID="8aa02c90926ede54901c9c3eadd9747a365f3232b402f928f59a9f8a3062fe65" Oct 31 00:58:49.315097 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:58:49.315070 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:59:00.508328 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:59:00.508274 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:59:19.319590 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:19.319551 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" podUID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400" Oct 31 00:59:29.317681 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:29.317637 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:59:30.509442 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:59:30.509398 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 00:59:37.588535 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:37.588505 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz"] Oct 31 00:59:37.589076 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:37.588744 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" podUID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" containerName="kserve-container" containerID="cri-o://d662dd6d82ae3440881896b7e45039736ae9d77d92dcfb656ce1d8c4dba39914" gracePeriod=30 Oct 31 00:59:37.589076 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:37.588818 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" podUID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" containerName="kube-rbac-proxy" containerID="cri-o://93987857d09b71357811921831ca8da2cacaafe59e92e46c2b4c9b6c46c9ca9b" gracePeriod=30 Oct 31 00:59:38.521808 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:38.521774 5939 generic.go:356] "Generic (PLEG): container finished" podID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" containerID="93987857d09b71357811921831ca8da2cacaafe59e92e46c2b4c9b6c46c9ca9b" exitCode=0 Oct 31 00:59:38.521999 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:38.521854 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" event={"ID":"30d818ac-ab96-45c6-a666-6dbd36d2ada2","Type":"ContainerDied","Data":"93987857d09b71357811921831ca8da2cacaafe59e92e46c2b4c9b6c46c9ca9b"} Oct 31 00:59:39.311420 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:39.311380 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" podUID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused" Oct 31 00:59:40.356878 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:40.356831 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" podUID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:59:44.311016 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:44.310966 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" podUID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused" Oct 31 00:59:44.930611 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:44.930586 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:59:45.064848 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.064765 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kube-rbac-proxy-sar-config\") pod \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " Oct 31 00:59:45.064848 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.064809 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d818ac-ab96-45c6-a666-6dbd36d2ada2-proxy-tls\" (UniqueName: \"kubernetes.io/secret/30d818ac-ab96-45c6-a666-6dbd36d2ada2-proxy-tls\") pod \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " Oct 31 00:59:45.064848 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.064833 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kube-api-access-npw7c\" (UniqueName: \"kubernetes.io/projected/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kube-api-access-npw7c\") pod \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " Oct 31 00:59:45.065101 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.064898 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kserve-provision-location\") pod \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " Oct 31 00:59:45.065101 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.064920 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d818ac-ab96-45c6-a666-6dbd36d2ada2-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30d818ac-ab96-45c6-a666-6dbd36d2ada2-cabundle-cert\") pod \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\" (UID: \"30d818ac-ab96-45c6-a666-6dbd36d2ada2\") " Oct 31 00:59:45.065264 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.065236 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kserve-provision-location" pod "30d818ac-ab96-45c6-a666-6dbd36d2ada2" (UID: "30d818ac-ab96-45c6-a666-6dbd36d2ada2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:45.065325 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.065286 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d818ac-ab96-45c6-a666-6dbd36d2ada2-cabundle-cert" pod "30d818ac-ab96-45c6-a666-6dbd36d2ada2" (UID: "30d818ac-ab96-45c6-a666-6dbd36d2ada2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:45.065384 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.065324 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:45.065384 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.065337 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kube-rbac-proxy-sar-config" pod "30d818ac-ab96-45c6-a666-6dbd36d2ada2" (UID: "30d818ac-ab96-45c6-a666-6dbd36d2ada2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:45.067089 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.067064 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kube-api-access-npw7c" pod "30d818ac-ab96-45c6-a666-6dbd36d2ada2" (UID: "30d818ac-ab96-45c6-a666-6dbd36d2ada2"). InnerVolumeSpecName "kube-api-access-npw7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:45.067089 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.067071 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d818ac-ab96-45c6-a666-6dbd36d2ada2-proxy-tls" pod "30d818ac-ab96-45c6-a666-6dbd36d2ada2" (UID: "30d818ac-ab96-45c6-a666-6dbd36d2ada2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:59:45.166545 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.166513 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30d818ac-ab96-45c6-a666-6dbd36d2ada2-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:45.166545 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.166539 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:45.166770 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.166571 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30d818ac-ab96-45c6-a666-6dbd36d2ada2-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:45.166770 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.166582 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npw7c\" (UniqueName: \"kubernetes.io/projected/30d818ac-ab96-45c6-a666-6dbd36d2ada2-kube-api-access-npw7c\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:45.550427 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.550395 5939 generic.go:356] "Generic (PLEG): container finished" podID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" containerID="d662dd6d82ae3440881896b7e45039736ae9d77d92dcfb656ce1d8c4dba39914" exitCode=0 Oct 31 00:59:45.550876 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.550456 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" Oct 31 00:59:45.550876 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.550474 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" event={"ID":"30d818ac-ab96-45c6-a666-6dbd36d2ada2","Type":"ContainerDied","Data":"d662dd6d82ae3440881896b7e45039736ae9d77d92dcfb656ce1d8c4dba39914"} Oct 31 00:59:45.550876 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.550534 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz" event={"ID":"30d818ac-ab96-45c6-a666-6dbd36d2ada2","Type":"ContainerDied","Data":"eb9f2b6ba245567a122855a7a989b4515a0d95d1d34c86cf146f51a30cceced4"} Oct 31 00:59:45.550876 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.550560 5939 scope.go:122] "RemoveContainer" containerID="93987857d09b71357811921831ca8da2cacaafe59e92e46c2b4c9b6c46c9ca9b" Oct 31 00:59:45.559800 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.559782 5939 scope.go:122] "RemoveContainer" containerID="d662dd6d82ae3440881896b7e45039736ae9d77d92dcfb656ce1d8c4dba39914" Oct 31 00:59:45.567067 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.567047 5939 scope.go:122] "RemoveContainer" containerID="a016de0dc8eac997e65d6b599ad75073b20178f1de817d1bb02e7ed0e427b2c6" Oct 31 00:59:45.574212 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.574196 5939 scope.go:122] "RemoveContainer" containerID="93987857d09b71357811921831ca8da2cacaafe59e92e46c2b4c9b6c46c9ca9b" Oct 31 00:59:45.574458 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:59:45.574437 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93987857d09b71357811921831ca8da2cacaafe59e92e46c2b4c9b6c46c9ca9b\": container with ID starting with 93987857d09b71357811921831ca8da2cacaafe59e92e46c2b4c9b6c46c9ca9b not found: ID does not exist" containerID="93987857d09b71357811921831ca8da2cacaafe59e92e46c2b4c9b6c46c9ca9b" Oct 31 00:59:45.574505 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.574467 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93987857d09b71357811921831ca8da2cacaafe59e92e46c2b4c9b6c46c9ca9b"} err="failed to get container status \"93987857d09b71357811921831ca8da2cacaafe59e92e46c2b4c9b6c46c9ca9b\": rpc error: code = NotFound desc = could not find container \"93987857d09b71357811921831ca8da2cacaafe59e92e46c2b4c9b6c46c9ca9b\": container with ID starting with 93987857d09b71357811921831ca8da2cacaafe59e92e46c2b4c9b6c46c9ca9b not found: ID does not exist" Oct 31 00:59:45.574505 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.574483 5939 scope.go:122] "RemoveContainer" containerID="d662dd6d82ae3440881896b7e45039736ae9d77d92dcfb656ce1d8c4dba39914" Oct 31 00:59:45.574700 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:59:45.574687 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d662dd6d82ae3440881896b7e45039736ae9d77d92dcfb656ce1d8c4dba39914\": container with ID starting with d662dd6d82ae3440881896b7e45039736ae9d77d92dcfb656ce1d8c4dba39914 not found: ID does not exist" containerID="d662dd6d82ae3440881896b7e45039736ae9d77d92dcfb656ce1d8c4dba39914" Oct 31 00:59:45.574746 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.574703 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d662dd6d82ae3440881896b7e45039736ae9d77d92dcfb656ce1d8c4dba39914"} err="failed to get container status \"d662dd6d82ae3440881896b7e45039736ae9d77d92dcfb656ce1d8c4dba39914\": rpc error: code = NotFound desc = could not find container \"d662dd6d82ae3440881896b7e45039736ae9d77d92dcfb656ce1d8c4dba39914\": container with ID starting with d662dd6d82ae3440881896b7e45039736ae9d77d92dcfb656ce1d8c4dba39914 not found: ID does not exist" Oct 31 00:59:45.574746 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.574717 5939 scope.go:122] "RemoveContainer" containerID="a016de0dc8eac997e65d6b599ad75073b20178f1de817d1bb02e7ed0e427b2c6" Oct 31 00:59:45.574920 ip-10-0-143-213 kubenswrapper[5939]: E1031 00:59:45.574906 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a016de0dc8eac997e65d6b599ad75073b20178f1de817d1bb02e7ed0e427b2c6\": container with ID starting with a016de0dc8eac997e65d6b599ad75073b20178f1de817d1bb02e7ed0e427b2c6 not found: ID does not exist" containerID="a016de0dc8eac997e65d6b599ad75073b20178f1de817d1bb02e7ed0e427b2c6" Oct 31 00:59:45.574959 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.574922 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a016de0dc8eac997e65d6b599ad75073b20178f1de817d1bb02e7ed0e427b2c6"} err="failed to get container status \"a016de0dc8eac997e65d6b599ad75073b20178f1de817d1bb02e7ed0e427b2c6\": rpc error: code = NotFound desc = could not find container \"a016de0dc8eac997e65d6b599ad75073b20178f1de817d1bb02e7ed0e427b2c6\": container with ID starting with a016de0dc8eac997e65d6b599ad75073b20178f1de817d1bb02e7ed0e427b2c6 not found: ID does not exist" Oct 31 00:59:45.582418 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.582397 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz"] Oct 31 00:59:45.589190 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.589170 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-runtime-predictor-675cd6cb7c-swjqz"] Oct 31 00:59:45.652322 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:45.652294 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" path="/var/lib/kubelet/pods/30d818ac-ab96-45c6-a666-6dbd36d2ada2/volumes" Oct 31 00:59:47.128977 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.128942 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb"] Oct 31 00:59:47.129361 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.129267 5939 cpu_manager.go:414] "containerMap: removing container" podUID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" containerName="storage-initializer" Oct 31 00:59:47.129361 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.129278 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" containerName="storage-initializer" Oct 31 00:59:47.129361 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.129292 5939 cpu_manager.go:414] "containerMap: removing container" podUID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" containerName="kserve-container" Oct 31 00:59:47.129361 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.129297 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" containerName="kserve-container" Oct 31 00:59:47.129361 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.129307 5939 cpu_manager.go:414] "containerMap: removing container" podUID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" containerName="kube-rbac-proxy" Oct 31 00:59:47.129361 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.129312 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" containerName="kube-rbac-proxy" Oct 31 00:59:47.129547 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.129378 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" containerName="kserve-container" Oct 31 00:59:47.129547 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.129386 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d818ac-ab96-45c6-a666-6dbd36d2ada2" containerName="kube-rbac-proxy" Oct 31 00:59:47.133756 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.133740 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:47.142269 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.142248 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\"" Oct 31 00:59:47.142387 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.142268 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 00:59:47.142387 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.142278 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 00:59:47.142387 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.142287 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 00:59:47.143210 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.143187 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 00:59:47.143345 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.143280 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-v2-predictor-serving-cert\"" Oct 31 00:59:47.143419 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.143381 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 00:59:47.146000 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.145979 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb"] Oct 31 00:59:47.286983 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.286946 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kts98\" (UniqueName: \"kubernetes.io/projected/2787884f-d81f-4799-b0c9-ad28607f1c3a-kube-api-access-kts98\") pod \"isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:47.287163 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.287002 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2787884f-d81f-4799-b0c9-ad28607f1c3a-proxy-tls\") pod \"isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:47.287163 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.287105 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2787884f-d81f-4799-b0c9-ad28607f1c3a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:47.287290 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.287164 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2787884f-d81f-4799-b0c9-ad28607f1c3a-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:47.287290 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.287264 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2787884f-d81f-4799-b0c9-ad28607f1c3a-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:47.387994 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.387905 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2787884f-d81f-4799-b0c9-ad28607f1c3a-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:47.387994 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.387945 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2787884f-d81f-4799-b0c9-ad28607f1c3a-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:47.387994 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.387978 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kts98\" (UniqueName: \"kubernetes.io/projected/2787884f-d81f-4799-b0c9-ad28607f1c3a-kube-api-access-kts98\") pod \"isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:47.388280 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.388007 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2787884f-d81f-4799-b0c9-ad28607f1c3a-proxy-tls\") pod \"isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:47.388280 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.388034 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2787884f-d81f-4799-b0c9-ad28607f1c3a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:47.388439 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.388405 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2787884f-d81f-4799-b0c9-ad28607f1c3a-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:47.388756 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.388731 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2787884f-d81f-4799-b0c9-ad28607f1c3a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:47.388934 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.388780 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2787884f-d81f-4799-b0c9-ad28607f1c3a-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:47.390598 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.390577 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2787884f-d81f-4799-b0c9-ad28607f1c3a-proxy-tls\") pod \"isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:47.402840 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.402815 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kts98\" (UniqueName: \"kubernetes.io/projected/2787884f-d81f-4799-b0c9-ad28607f1c3a-kube-api-access-kts98\") pod \"isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:47.452847 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.452810 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:47.733070 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.732987 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb"] Oct 31 00:59:47.735943 ip-10-0-143-213 kubenswrapper[5939]: W1031 00:59:47.735914 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2787884f_d81f_4799_b0c9_ad28607f1c3a.slice/crio-b92a2d9ebf68b43476dcaa04a80a0d61004e73c3aedcc0de62ea02b99eeeae63 WatchSource:0}: Error finding container b92a2d9ebf68b43476dcaa04a80a0d61004e73c3aedcc0de62ea02b99eeeae63: Status 404 returned error can't find the container with id b92a2d9ebf68b43476dcaa04a80a0d61004e73c3aedcc0de62ea02b99eeeae63 Oct 31 00:59:47.741137 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.741115 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 00:59:47.862907 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.862879 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:59:47.866223 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:47.866201 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 00:59:48.565558 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:48.565516 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" event={"ID":"2787884f-d81f-4799-b0c9-ad28607f1c3a","Type":"ContainerStarted","Data":"aabf526c09423319c28a57947162d7e86d15dbbe017d4a91b00eeedae1e4b0ea"} Oct 31 00:59:48.565967 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:48.565626 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" event={"ID":"2787884f-d81f-4799-b0c9-ad28607f1c3a","Type":"ContainerStarted","Data":"b92a2d9ebf68b43476dcaa04a80a0d61004e73c3aedcc0de62ea02b99eeeae63"} Oct 31 00:59:52.584227 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:52.584192 5939 generic.go:356] "Generic (PLEG): container finished" podID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerID="aabf526c09423319c28a57947162d7e86d15dbbe017d4a91b00eeedae1e4b0ea" exitCode=0 Oct 31 00:59:52.584649 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:52.584247 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" event={"ID":"2787884f-d81f-4799-b0c9-ad28607f1c3a","Type":"ContainerDied","Data":"aabf526c09423319c28a57947162d7e86d15dbbe017d4a91b00eeedae1e4b0ea"} Oct 31 00:59:53.590395 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:53.590355 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" event={"ID":"2787884f-d81f-4799-b0c9-ad28607f1c3a","Type":"ContainerStarted","Data":"6efae7a366dfcb5fd32c11d2505c09ef1e2b7e146d77d88a9dbf3fed404b6538"} Oct 31 00:59:53.590825 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:53.590446 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" event={"ID":"2787884f-d81f-4799-b0c9-ad28607f1c3a","Type":"ContainerStarted","Data":"358ae6b6d0577927de379f0a7ca734d0bdfda0b743f82507b68e1c07c436e609"} Oct 31 00:59:53.590825 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:53.590785 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:53.590903 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:53.590841 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:53.591980 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:53.591953 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Oct 31 00:59:53.620677 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:53.620627 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" podStartSLOduration=2.792237903 podStartE2EDuration="6.620614717s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.828376814s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:53.618620035 +0000 UTC m=+1806.107895216" watchObservedRunningTime="2026-10-31 00:59:53.620614717 +0000 UTC m=+1806.109889895" Oct 31 00:59:54.594893 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:54.594857 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Oct 31 00:59:59.599907 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:59.599835 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 00:59:59.600373 ip-10-0-143-213 kubenswrapper[5939]: I1031 00:59:59.600347 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Oct 31 01:00:00.509162 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:00:00.509121 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:00:09.600445 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:00:09.600399 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Oct 31 01:00:19.600989 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:00:19.600947 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Oct 31 01:00:29.600660 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:00:29.600618 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Oct 31 01:00:30.509402 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:00:30.509367 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:00:39.600922 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:00:39.600879 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Oct 31 01:00:49.600246 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:00:49.600197 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Oct 31 01:00:59.601160 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:00:59.601115 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Oct 31 01:01:00.508396 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:01:00.508361 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:01:09.601650 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:09.601617 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 01:01:18.236364 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:18.236322 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb"] Oct 31 01:01:18.236910 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:18.236622 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kserve-container" containerID="cri-o://358ae6b6d0577927de379f0a7ca734d0bdfda0b743f82507b68e1c07c436e609" gracePeriod=30 Oct 31 01:01:18.236910 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:18.236643 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kube-rbac-proxy" containerID="cri-o://6efae7a366dfcb5fd32c11d2505c09ef1e2b7e146d77d88a9dbf3fed404b6538" gracePeriod=30 Oct 31 01:01:18.927067 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:18.927027 5939 generic.go:356] "Generic (PLEG): container finished" podID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerID="6efae7a366dfcb5fd32c11d2505c09ef1e2b7e146d77d88a9dbf3fed404b6538" exitCode=0 Oct 31 01:01:18.927242 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:18.927101 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" event={"ID":"2787884f-d81f-4799-b0c9-ad28607f1c3a","Type":"ContainerDied","Data":"6efae7a366dfcb5fd32c11d2505c09ef1e2b7e146d77d88a9dbf3fed404b6538"} Oct 31 01:01:19.595707 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:19.595644 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused" Oct 31 01:01:19.601183 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:19.601157 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Oct 31 01:01:23.379441 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.379413 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 01:01:23.512471 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.512369 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2787884f-d81f-4799-b0c9-ad28607f1c3a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2787884f-d81f-4799-b0c9-ad28607f1c3a-cabundle-cert\") pod \"2787884f-d81f-4799-b0c9-ad28607f1c3a\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " Oct 31 01:01:23.512471 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.512439 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2787884f-d81f-4799-b0c9-ad28607f1c3a-kube-api-access-kts98\" (UniqueName: \"kubernetes.io/projected/2787884f-d81f-4799-b0c9-ad28607f1c3a-kube-api-access-kts98\") pod \"2787884f-d81f-4799-b0c9-ad28607f1c3a\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " Oct 31 01:01:23.512748 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.512497 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2787884f-d81f-4799-b0c9-ad28607f1c3a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2787884f-d81f-4799-b0c9-ad28607f1c3a-kube-rbac-proxy-sar-config\") pod \"2787884f-d81f-4799-b0c9-ad28607f1c3a\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " Oct 31 01:01:23.512748 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.512544 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2787884f-d81f-4799-b0c9-ad28607f1c3a-proxy-tls\" (UniqueName: \"kubernetes.io/secret/2787884f-d81f-4799-b0c9-ad28607f1c3a-proxy-tls\") pod \"2787884f-d81f-4799-b0c9-ad28607f1c3a\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " Oct 31 01:01:23.512748 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.512606 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2787884f-d81f-4799-b0c9-ad28607f1c3a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2787884f-d81f-4799-b0c9-ad28607f1c3a-kserve-provision-location\") pod \"2787884f-d81f-4799-b0c9-ad28607f1c3a\" (UID: \"2787884f-d81f-4799-b0c9-ad28607f1c3a\") " Oct 31 01:01:23.512925 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.512899 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2787884f-d81f-4799-b0c9-ad28607f1c3a-cabundle-cert" pod "2787884f-d81f-4799-b0c9-ad28607f1c3a" (UID: "2787884f-d81f-4799-b0c9-ad28607f1c3a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:23.512988 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.512938 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2787884f-d81f-4799-b0c9-ad28607f1c3a-kserve-provision-location" pod "2787884f-d81f-4799-b0c9-ad28607f1c3a" (UID: "2787884f-d81f-4799-b0c9-ad28607f1c3a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:23.513045 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.513009 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2787884f-d81f-4799-b0c9-ad28607f1c3a-kube-rbac-proxy-sar-config" pod "2787884f-d81f-4799-b0c9-ad28607f1c3a" (UID: "2787884f-d81f-4799-b0c9-ad28607f1c3a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:23.514602 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.514584 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2787884f-d81f-4799-b0c9-ad28607f1c3a-proxy-tls" pod "2787884f-d81f-4799-b0c9-ad28607f1c3a" (UID: "2787884f-d81f-4799-b0c9-ad28607f1c3a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:23.514683 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.514651 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2787884f-d81f-4799-b0c9-ad28607f1c3a-kube-api-access-kts98" pod "2787884f-d81f-4799-b0c9-ad28607f1c3a" (UID: "2787884f-d81f-4799-b0c9-ad28607f1c3a"). InnerVolumeSpecName "kube-api-access-kts98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:23.614096 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.614057 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2787884f-d81f-4799-b0c9-ad28607f1c3a-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:23.614096 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.614091 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kts98\" (UniqueName: \"kubernetes.io/projected/2787884f-d81f-4799-b0c9-ad28607f1c3a-kube-api-access-kts98\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:23.614096 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.614104 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2787884f-d81f-4799-b0c9-ad28607f1c3a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:23.614334 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.614113 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2787884f-d81f-4799-b0c9-ad28607f1c3a-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:23.614334 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.614122 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2787884f-d81f-4799-b0c9-ad28607f1c3a-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:23.948814 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.948777 5939 generic.go:356] "Generic (PLEG): container finished" podID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerID="358ae6b6d0577927de379f0a7ca734d0bdfda0b743f82507b68e1c07c436e609" exitCode=0 Oct 31 01:01:23.948979 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.948828 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" event={"ID":"2787884f-d81f-4799-b0c9-ad28607f1c3a","Type":"ContainerDied","Data":"358ae6b6d0577927de379f0a7ca734d0bdfda0b743f82507b68e1c07c436e609"} Oct 31 01:01:23.948979 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.948848 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" Oct 31 01:01:23.948979 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.948893 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb" event={"ID":"2787884f-d81f-4799-b0c9-ad28607f1c3a","Type":"ContainerDied","Data":"b92a2d9ebf68b43476dcaa04a80a0d61004e73c3aedcc0de62ea02b99eeeae63"} Oct 31 01:01:23.948979 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.948929 5939 scope.go:122] "RemoveContainer" containerID="6efae7a366dfcb5fd32c11d2505c09ef1e2b7e146d77d88a9dbf3fed404b6538" Oct 31 01:01:23.956872 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.956851 5939 scope.go:122] "RemoveContainer" containerID="358ae6b6d0577927de379f0a7ca734d0bdfda0b743f82507b68e1c07c436e609" Oct 31 01:01:23.964786 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.964766 5939 scope.go:122] "RemoveContainer" containerID="aabf526c09423319c28a57947162d7e86d15dbbe017d4a91b00eeedae1e4b0ea" Oct 31 01:01:23.972556 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.972513 5939 scope.go:122] "RemoveContainer" containerID="6efae7a366dfcb5fd32c11d2505c09ef1e2b7e146d77d88a9dbf3fed404b6538" Oct 31 01:01:23.972915 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:01:23.972889 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6efae7a366dfcb5fd32c11d2505c09ef1e2b7e146d77d88a9dbf3fed404b6538\": container with ID starting with 6efae7a366dfcb5fd32c11d2505c09ef1e2b7e146d77d88a9dbf3fed404b6538 not found: ID does not exist" containerID="6efae7a366dfcb5fd32c11d2505c09ef1e2b7e146d77d88a9dbf3fed404b6538" Oct 31 01:01:23.973037 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.972926 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6efae7a366dfcb5fd32c11d2505c09ef1e2b7e146d77d88a9dbf3fed404b6538"} err="failed to get container status \"6efae7a366dfcb5fd32c11d2505c09ef1e2b7e146d77d88a9dbf3fed404b6538\": rpc error: code = NotFound desc = could not find container \"6efae7a366dfcb5fd32c11d2505c09ef1e2b7e146d77d88a9dbf3fed404b6538\": container with ID starting with 6efae7a366dfcb5fd32c11d2505c09ef1e2b7e146d77d88a9dbf3fed404b6538 not found: ID does not exist" Oct 31 01:01:23.973037 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.972952 5939 scope.go:122] "RemoveContainer" containerID="358ae6b6d0577927de379f0a7ca734d0bdfda0b743f82507b68e1c07c436e609" Oct 31 01:01:23.973365 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:01:23.973341 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358ae6b6d0577927de379f0a7ca734d0bdfda0b743f82507b68e1c07c436e609\": container with ID starting with 358ae6b6d0577927de379f0a7ca734d0bdfda0b743f82507b68e1c07c436e609 not found: ID does not exist" containerID="358ae6b6d0577927de379f0a7ca734d0bdfda0b743f82507b68e1c07c436e609" Oct 31 01:01:23.973449 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.973381 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358ae6b6d0577927de379f0a7ca734d0bdfda0b743f82507b68e1c07c436e609"} err="failed to get container status \"358ae6b6d0577927de379f0a7ca734d0bdfda0b743f82507b68e1c07c436e609\": rpc error: code = NotFound desc = could not find container \"358ae6b6d0577927de379f0a7ca734d0bdfda0b743f82507b68e1c07c436e609\": container with ID starting with 358ae6b6d0577927de379f0a7ca734d0bdfda0b743f82507b68e1c07c436e609 not found: ID does not exist" Oct 31 01:01:23.973449 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.973402 5939 scope.go:122] "RemoveContainer" containerID="aabf526c09423319c28a57947162d7e86d15dbbe017d4a91b00eeedae1e4b0ea" Oct 31 01:01:23.973746 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:01:23.973724 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aabf526c09423319c28a57947162d7e86d15dbbe017d4a91b00eeedae1e4b0ea\": container with ID starting with aabf526c09423319c28a57947162d7e86d15dbbe017d4a91b00eeedae1e4b0ea not found: ID does not exist" containerID="aabf526c09423319c28a57947162d7e86d15dbbe017d4a91b00eeedae1e4b0ea" Oct 31 01:01:23.973838 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.973751 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aabf526c09423319c28a57947162d7e86d15dbbe017d4a91b00eeedae1e4b0ea"} err="failed to get container status \"aabf526c09423319c28a57947162d7e86d15dbbe017d4a91b00eeedae1e4b0ea\": rpc error: code = NotFound desc = could not find container \"aabf526c09423319c28a57947162d7e86d15dbbe017d4a91b00eeedae1e4b0ea\": container with ID starting with aabf526c09423319c28a57947162d7e86d15dbbe017d4a91b00eeedae1e4b0ea not found: ID does not exist" Oct 31 01:01:23.974382 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.974360 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb"] Oct 31 01:01:23.980294 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:23.980265 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-predictor-dfcb46dc9-tr8bb"] Oct 31 01:01:25.651418 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.651386 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" path="/var/lib/kubelet/pods/2787884f-d81f-4799-b0c9-ad28607f1c3a/volumes" Oct 31 01:01:25.727567 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.727534 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8"] Oct 31 01:01:25.727875 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.727861 5939 cpu_manager.go:414] "containerMap: removing container" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="storage-initializer" Oct 31 01:01:25.727935 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.727875 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="storage-initializer" Oct 31 01:01:25.727935 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.727893 5939 cpu_manager.go:414] "containerMap: removing container" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kserve-container" Oct 31 01:01:25.727935 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.727899 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kserve-container" Oct 31 01:01:25.727935 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.727923 5939 cpu_manager.go:414] "containerMap: removing container" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kube-rbac-proxy" Oct 31 01:01:25.727935 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.727929 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kube-rbac-proxy" Oct 31 01:01:25.728086 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.727983 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kserve-container" Oct 31 01:01:25.728086 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.727995 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="2787884f-d81f-4799-b0c9-ad28607f1c3a" containerName="kube-rbac-proxy" Oct 31 01:01:25.732636 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.732618 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:25.739344 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.739070 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 01:01:25.739344 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.739130 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\"" Oct 31 01:01:25.739344 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.739215 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\"" Oct 31 01:01:25.739344 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.739274 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 01:01:25.739344 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.739280 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 01:01:25.739344 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.739322 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 01:01:25.739344 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.739324 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 01:01:25.746453 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.746425 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8"] Oct 31 01:01:25.836231 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.836199 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f84433b2-4aac-44d6-ae7e-f67688bd2812-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:25.836449 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.836250 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f84433b2-4aac-44d6-ae7e-f67688bd2812-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:25.836449 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.836385 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f84433b2-4aac-44d6-ae7e-f67688bd2812-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:25.836449 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.836416 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krwgf\" (UniqueName: \"kubernetes.io/projected/f84433b2-4aac-44d6-ae7e-f67688bd2812-kube-api-access-krwgf\") pod \"isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:25.836571 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.836447 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f84433b2-4aac-44d6-ae7e-f67688bd2812-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:25.937414 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.937312 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f84433b2-4aac-44d6-ae7e-f67688bd2812-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:25.937414 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.937369 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f84433b2-4aac-44d6-ae7e-f67688bd2812-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:25.937414 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.937419 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f84433b2-4aac-44d6-ae7e-f67688bd2812-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:25.937730 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.937444 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krwgf\" (UniqueName: \"kubernetes.io/projected/f84433b2-4aac-44d6-ae7e-f67688bd2812-kube-api-access-krwgf\") pod \"isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:25.937730 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.937466 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f84433b2-4aac-44d6-ae7e-f67688bd2812-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:25.937923 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.937896 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f84433b2-4aac-44d6-ae7e-f67688bd2812-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:25.938211 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.938191 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f84433b2-4aac-44d6-ae7e-f67688bd2812-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:25.938258 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.938202 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f84433b2-4aac-44d6-ae7e-f67688bd2812-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:25.939937 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.939912 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f84433b2-4aac-44d6-ae7e-f67688bd2812-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:25.952073 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:25.952049 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krwgf\" (UniqueName: \"kubernetes.io/projected/f84433b2-4aac-44d6-ae7e-f67688bd2812-kube-api-access-krwgf\") pod \"isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:26.056789 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:26.056756 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:26.337783 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:26.337587 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8"] Oct 31 01:01:26.340600 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:01:26.340569 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84433b2_4aac_44d6_ae7e_f67688bd2812.slice/crio-12a286371aee140158504b67fe08514a9e8766081e6ebbabc3acc28a99c77322 WatchSource:0}: Error finding container 12a286371aee140158504b67fe08514a9e8766081e6ebbabc3acc28a99c77322: Status 404 returned error can't find the container with id 12a286371aee140158504b67fe08514a9e8766081e6ebbabc3acc28a99c77322 Oct 31 01:01:26.962190 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:26.962155 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" event={"ID":"f84433b2-4aac-44d6-ae7e-f67688bd2812","Type":"ContainerStarted","Data":"cb45187b8216c023b88b71ab30f90e3ee314cf6987fb1032e9989162d6704550"} Oct 31 01:01:26.962686 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:26.962224 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" event={"ID":"f84433b2-4aac-44d6-ae7e-f67688bd2812","Type":"ContainerStarted","Data":"12a286371aee140158504b67fe08514a9e8766081e6ebbabc3acc28a99c77322"} Oct 31 01:01:30.508705 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:01:30.508613 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:01:30.979732 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:30.979698 5939 generic.go:356] "Generic (PLEG): container finished" podID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerID="cb45187b8216c023b88b71ab30f90e3ee314cf6987fb1032e9989162d6704550" exitCode=0 Oct 31 01:01:30.979940 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:30.979762 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" event={"ID":"f84433b2-4aac-44d6-ae7e-f67688bd2812","Type":"ContainerDied","Data":"cb45187b8216c023b88b71ab30f90e3ee314cf6987fb1032e9989162d6704550"} Oct 31 01:01:31.987228 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:31.987186 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" event={"ID":"f84433b2-4aac-44d6-ae7e-f67688bd2812","Type":"ContainerStarted","Data":"e11b74721108d5d58ac9a7b28b17b6d815727284924cd47b41be757328804e7b"} Oct 31 01:01:31.987615 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:31.987275 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" event={"ID":"f84433b2-4aac-44d6-ae7e-f67688bd2812","Type":"ContainerStarted","Data":"bdb2c59a9478fabd2322544a25da2d44083c188eafc4107b1f360a33236e3acf"} Oct 31 01:01:31.987615 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:31.987507 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:32.015752 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:32.015692 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" podStartSLOduration=3.157553637 podStartE2EDuration="7.015678563s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.858124926s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:32.015332332 +0000 UTC m=+1904.504607513" watchObservedRunningTime="2026-10-31 01:01:32.015678563 +0000 UTC m=+1904.504953733" Oct 31 01:01:32.991940 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:32.991906 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:32.993264 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:32.993232 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Oct 31 01:01:33.995729 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:33.995689 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Oct 31 01:01:38.999770 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:38.999741 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:01:39.000202 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:39.000173 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Oct 31 01:01:49.000725 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:49.000684 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Oct 31 01:01:59.000585 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:01:59.000544 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Oct 31 01:02:00.509534 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:02:00.509497 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:02:09.000181 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:02:09.000132 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Oct 31 01:02:19.000252 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:02:19.000205 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Oct 31 01:02:29.000921 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:02:29.000882 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Oct 31 01:02:30.508207 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:02:30.508169 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:02:39.000299 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:02:39.000255 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Oct 31 01:02:49.000682 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:02:49.000645 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:02:56.806558 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:02:56.806511 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8"] Oct 31 01:02:56.807082 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:02:56.806815 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kserve-container" containerID="cri-o://bdb2c59a9478fabd2322544a25da2d44083c188eafc4107b1f360a33236e3acf" gracePeriod=30 Oct 31 01:02:56.807082 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:02:56.806852 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kube-rbac-proxy" containerID="cri-o://e11b74721108d5d58ac9a7b28b17b6d815727284924cd47b41be757328804e7b" gracePeriod=30 Oct 31 01:02:57.329008 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:02:57.328974 5939 generic.go:356] "Generic (PLEG): container finished" podID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerID="e11b74721108d5d58ac9a7b28b17b6d815727284924cd47b41be757328804e7b" exitCode=0 Oct 31 01:02:57.329177 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:02:57.329050 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" event={"ID":"f84433b2-4aac-44d6-ae7e-f67688bd2812","Type":"ContainerDied","Data":"e11b74721108d5d58ac9a7b28b17b6d815727284924cd47b41be757328804e7b"} Oct 31 01:02:58.996981 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:02:58.996896 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused" Oct 31 01:02:59.000197 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:02:59.000172 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Oct 31 01:03:00.509101 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:03:00.509067 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:03:01.948487 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:01.948461 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:03:02.078398 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.078316 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84433b2-4aac-44d6-ae7e-f67688bd2812-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f84433b2-4aac-44d6-ae7e-f67688bd2812-kube-rbac-proxy-sar-config\") pod \"f84433b2-4aac-44d6-ae7e-f67688bd2812\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " Oct 31 01:03:02.078539 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.078406 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f84433b2-4aac-44d6-ae7e-f67688bd2812-proxy-tls\" (UniqueName: \"kubernetes.io/secret/f84433b2-4aac-44d6-ae7e-f67688bd2812-proxy-tls\") pod \"f84433b2-4aac-44d6-ae7e-f67688bd2812\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " Oct 31 01:03:02.078539 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.078451 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84433b2-4aac-44d6-ae7e-f67688bd2812-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f84433b2-4aac-44d6-ae7e-f67688bd2812-kserve-provision-location\") pod \"f84433b2-4aac-44d6-ae7e-f67688bd2812\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " Oct 31 01:03:02.078539 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.078487 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84433b2-4aac-44d6-ae7e-f67688bd2812-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f84433b2-4aac-44d6-ae7e-f67688bd2812-cabundle-cert\") pod \"f84433b2-4aac-44d6-ae7e-f67688bd2812\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " Oct 31 01:03:02.078539 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.078519 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84433b2-4aac-44d6-ae7e-f67688bd2812-kube-api-access-krwgf\" (UniqueName: \"kubernetes.io/projected/f84433b2-4aac-44d6-ae7e-f67688bd2812-kube-api-access-krwgf\") pod \"f84433b2-4aac-44d6-ae7e-f67688bd2812\" (UID: \"f84433b2-4aac-44d6-ae7e-f67688bd2812\") " Oct 31 01:03:02.078768 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.078738 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84433b2-4aac-44d6-ae7e-f67688bd2812-kube-rbac-proxy-sar-config" pod "f84433b2-4aac-44d6-ae7e-f67688bd2812" (UID: "f84433b2-4aac-44d6-ae7e-f67688bd2812"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:02.078850 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.078823 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84433b2-4aac-44d6-ae7e-f67688bd2812-kserve-provision-location" pod "f84433b2-4aac-44d6-ae7e-f67688bd2812" (UID: "f84433b2-4aac-44d6-ae7e-f67688bd2812"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:02.078919 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.078872 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84433b2-4aac-44d6-ae7e-f67688bd2812-cabundle-cert" pod "f84433b2-4aac-44d6-ae7e-f67688bd2812" (UID: "f84433b2-4aac-44d6-ae7e-f67688bd2812"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:02.080455 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.080434 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84433b2-4aac-44d6-ae7e-f67688bd2812-proxy-tls" pod "f84433b2-4aac-44d6-ae7e-f67688bd2812" (UID: "f84433b2-4aac-44d6-ae7e-f67688bd2812"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:03:02.080580 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.080557 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84433b2-4aac-44d6-ae7e-f67688bd2812-kube-api-access-krwgf" pod "f84433b2-4aac-44d6-ae7e-f67688bd2812" (UID: "f84433b2-4aac-44d6-ae7e-f67688bd2812"). InnerVolumeSpecName "kube-api-access-krwgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:02.179428 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.179391 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f84433b2-4aac-44d6-ae7e-f67688bd2812-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:02.179428 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.179423 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f84433b2-4aac-44d6-ae7e-f67688bd2812-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:02.179428 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.179435 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f84433b2-4aac-44d6-ae7e-f67688bd2812-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:02.179692 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.179450 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krwgf\" (UniqueName: \"kubernetes.io/projected/f84433b2-4aac-44d6-ae7e-f67688bd2812-kube-api-access-krwgf\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:02.179692 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.179462 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f84433b2-4aac-44d6-ae7e-f67688bd2812-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:02.350435 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.350347 5939 generic.go:356] "Generic (PLEG): container finished" podID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerID="bdb2c59a9478fabd2322544a25da2d44083c188eafc4107b1f360a33236e3acf" exitCode=0 Oct 31 01:03:02.350435 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.350421 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" Oct 31 01:03:02.350435 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.350425 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" event={"ID":"f84433b2-4aac-44d6-ae7e-f67688bd2812","Type":"ContainerDied","Data":"bdb2c59a9478fabd2322544a25da2d44083c188eafc4107b1f360a33236e3acf"} Oct 31 01:03:02.350688 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.350511 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8" event={"ID":"f84433b2-4aac-44d6-ae7e-f67688bd2812","Type":"ContainerDied","Data":"12a286371aee140158504b67fe08514a9e8766081e6ebbabc3acc28a99c77322"} Oct 31 01:03:02.350688 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.350540 5939 scope.go:122] "RemoveContainer" containerID="e11b74721108d5d58ac9a7b28b17b6d815727284924cd47b41be757328804e7b" Oct 31 01:03:02.358987 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.358973 5939 scope.go:122] "RemoveContainer" containerID="bdb2c59a9478fabd2322544a25da2d44083c188eafc4107b1f360a33236e3acf" Oct 31 01:03:02.366354 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.366338 5939 scope.go:122] "RemoveContainer" containerID="cb45187b8216c023b88b71ab30f90e3ee314cf6987fb1032e9989162d6704550" Oct 31 01:03:02.373503 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.373485 5939 scope.go:122] "RemoveContainer" containerID="e11b74721108d5d58ac9a7b28b17b6d815727284924cd47b41be757328804e7b" Oct 31 01:03:02.373816 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:03:02.373788 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11b74721108d5d58ac9a7b28b17b6d815727284924cd47b41be757328804e7b\": container with ID starting with e11b74721108d5d58ac9a7b28b17b6d815727284924cd47b41be757328804e7b not found: ID does not exist" containerID="e11b74721108d5d58ac9a7b28b17b6d815727284924cd47b41be757328804e7b" Oct 31 01:03:02.373905 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.373825 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11b74721108d5d58ac9a7b28b17b6d815727284924cd47b41be757328804e7b"} err="failed to get container status \"e11b74721108d5d58ac9a7b28b17b6d815727284924cd47b41be757328804e7b\": rpc error: code = NotFound desc = could not find container \"e11b74721108d5d58ac9a7b28b17b6d815727284924cd47b41be757328804e7b\": container with ID starting with e11b74721108d5d58ac9a7b28b17b6d815727284924cd47b41be757328804e7b not found: ID does not exist" Oct 31 01:03:02.373905 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.373849 5939 scope.go:122] "RemoveContainer" containerID="bdb2c59a9478fabd2322544a25da2d44083c188eafc4107b1f360a33236e3acf" Oct 31 01:03:02.374155 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:03:02.374129 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb2c59a9478fabd2322544a25da2d44083c188eafc4107b1f360a33236e3acf\": container with ID starting with bdb2c59a9478fabd2322544a25da2d44083c188eafc4107b1f360a33236e3acf not found: ID does not exist" containerID="bdb2c59a9478fabd2322544a25da2d44083c188eafc4107b1f360a33236e3acf" Oct 31 01:03:02.374274 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.374159 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb2c59a9478fabd2322544a25da2d44083c188eafc4107b1f360a33236e3acf"} err="failed to get container status \"bdb2c59a9478fabd2322544a25da2d44083c188eafc4107b1f360a33236e3acf\": rpc error: code = NotFound desc = could not find container \"bdb2c59a9478fabd2322544a25da2d44083c188eafc4107b1f360a33236e3acf\": container with ID starting with bdb2c59a9478fabd2322544a25da2d44083c188eafc4107b1f360a33236e3acf not found: ID does not exist" Oct 31 01:03:02.374274 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.374175 5939 scope.go:122] "RemoveContainer" containerID="cb45187b8216c023b88b71ab30f90e3ee314cf6987fb1032e9989162d6704550" Oct 31 01:03:02.374486 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:03:02.374466 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb45187b8216c023b88b71ab30f90e3ee314cf6987fb1032e9989162d6704550\": container with ID starting with cb45187b8216c023b88b71ab30f90e3ee314cf6987fb1032e9989162d6704550 not found: ID does not exist" containerID="cb45187b8216c023b88b71ab30f90e3ee314cf6987fb1032e9989162d6704550" Oct 31 01:03:02.374553 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.374495 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb45187b8216c023b88b71ab30f90e3ee314cf6987fb1032e9989162d6704550"} err="failed to get container status \"cb45187b8216c023b88b71ab30f90e3ee314cf6987fb1032e9989162d6704550\": rpc error: code = NotFound desc = could not find container \"cb45187b8216c023b88b71ab30f90e3ee314cf6987fb1032e9989162d6704550\": container with ID starting with cb45187b8216c023b88b71ab30f90e3ee314cf6987fb1032e9989162d6704550 not found: ID does not exist" Oct 31 01:03:02.375763 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.375743 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8"] Oct 31 01:03:02.382001 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:02.381980 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-v2-mixed-predictor-6878dbd9c4-bn9x8"] Oct 31 01:03:03.652403 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:03.652370 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" path="/var/lib/kubelet/pods/f84433b2-4aac-44d6-ae7e-f67688bd2812/volumes" Oct 31 01:03:06.064481 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.064445 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245"] Oct 31 01:03:06.064876 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.064811 5939 cpu_manager.go:414] "containerMap: removing container" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kube-rbac-proxy" Oct 31 01:03:06.064876 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.064822 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kube-rbac-proxy" Oct 31 01:03:06.064876 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.064846 5939 cpu_manager.go:414] "containerMap: removing container" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="storage-initializer" Oct 31 01:03:06.064876 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.064852 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="storage-initializer" Oct 31 01:03:06.064876 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.064858 5939 cpu_manager.go:414] "containerMap: removing container" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kserve-container" Oct 31 01:03:06.064876 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.064865 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kserve-container" Oct 31 01:03:06.065074 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.064921 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kserve-container" Oct 31 01:03:06.065074 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.064933 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84433b2-4aac-44d6-ae7e-f67688bd2812" containerName="kube-rbac-proxy" Oct 31 01:03:06.069719 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.069701 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:03:06.075542 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.075506 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\"" Oct 31 01:03:06.076814 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.076792 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 01:03:06.076814 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.076803 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\"" Oct 31 01:03:06.076987 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.076803 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\"" Oct 31 01:03:06.076987 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.076821 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\"" Oct 31 01:03:06.076987 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.076810 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\"" Oct 31 01:03:06.076987 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.076875 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\"" Oct 31 01:03:06.081791 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.081770 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245"] Oct 31 01:03:06.215653 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.215620 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5386e312-8c91-4461-b8b9-35b06185b5b8-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5694484bf-jj245\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:03:06.215877 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.215681 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5386e312-8c91-4461-b8b9-35b06185b5b8-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5694484bf-jj245\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:03:06.215877 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.215818 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5386e312-8c91-4461-b8b9-35b06185b5b8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5694484bf-jj245\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:03:06.216018 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.215944 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx4rm\" (UniqueName: \"kubernetes.io/projected/5386e312-8c91-4461-b8b9-35b06185b5b8-kube-api-access-kx4rm\") pod \"isvc-autogluon-v2-variants-predictor-5694484bf-jj245\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:03:06.216018 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.215987 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5386e312-8c91-4461-b8b9-35b06185b5b8-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5694484bf-jj245\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:03:06.316692 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.316606 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx4rm\" (UniqueName: \"kubernetes.io/projected/5386e312-8c91-4461-b8b9-35b06185b5b8-kube-api-access-kx4rm\") pod \"isvc-autogluon-v2-variants-predictor-5694484bf-jj245\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:03:06.316692 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.316640 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5386e312-8c91-4461-b8b9-35b06185b5b8-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5694484bf-jj245\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:03:06.316692 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.316663 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5386e312-8c91-4461-b8b9-35b06185b5b8-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5694484bf-jj245\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:03:06.316951 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.316834 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5386e312-8c91-4461-b8b9-35b06185b5b8-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5694484bf-jj245\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:03:06.316951 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.316882 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5386e312-8c91-4461-b8b9-35b06185b5b8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5694484bf-jj245\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:03:06.317171 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.317152 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5386e312-8c91-4461-b8b9-35b06185b5b8-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5694484bf-jj245\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:03:06.317376 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.317358 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5386e312-8c91-4461-b8b9-35b06185b5b8-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5694484bf-jj245\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:03:06.317505 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.317486 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5386e312-8c91-4461-b8b9-35b06185b5b8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5694484bf-jj245\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:03:06.319201 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.319172 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5386e312-8c91-4461-b8b9-35b06185b5b8-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5694484bf-jj245\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:03:06.331287 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.331266 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx4rm\" (UniqueName: \"kubernetes.io/projected/5386e312-8c91-4461-b8b9-35b06185b5b8-kube-api-access-kx4rm\") pod \"isvc-autogluon-v2-variants-predictor-5694484bf-jj245\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:03:06.387075 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.387046 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:03:06.667747 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:06.667715 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245"] Oct 31 01:03:06.671188 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:03:06.671161 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5386e312_8c91_4461_b8b9_35b06185b5b8.slice/crio-d5a31f4f7d5b444ed8a6c6061298819e5514d53b18060d79f80c3872fdbfcbdb WatchSource:0}: Error finding container d5a31f4f7d5b444ed8a6c6061298819e5514d53b18060d79f80c3872fdbfcbdb: Status 404 returned error can't find the container with id d5a31f4f7d5b444ed8a6c6061298819e5514d53b18060d79f80c3872fdbfcbdb Oct 31 01:03:07.373534 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:07.373496 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" event={"ID":"5386e312-8c91-4461-b8b9-35b06185b5b8","Type":"ContainerStarted","Data":"19ab5ef695aac27e74fbf5a232349423406ff87a2abf4b54f9adf1edb75a3d0a"} Oct 31 01:03:07.374031 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:07.373555 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" event={"ID":"5386e312-8c91-4461-b8b9-35b06185b5b8","Type":"ContainerStarted","Data":"d5a31f4f7d5b444ed8a6c6061298819e5514d53b18060d79f80c3872fdbfcbdb"} Oct 31 01:03:19.423268 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:19.423232 5939 generic.go:356] "Generic (PLEG): container finished" podID="5386e312-8c91-4461-b8b9-35b06185b5b8" containerID="19ab5ef695aac27e74fbf5a232349423406ff87a2abf4b54f9adf1edb75a3d0a" exitCode=0 Oct 31 01:03:19.423662 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:19.423306 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" event={"ID":"5386e312-8c91-4461-b8b9-35b06185b5b8","Type":"ContainerDied","Data":"19ab5ef695aac27e74fbf5a232349423406ff87a2abf4b54f9adf1edb75a3d0a"} Oct 31 01:03:19.424611 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:03:19.424595 5939 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:03:30.517172 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:03:30.517125 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:04:00.511905 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:04:00.511861 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:04:30.578896 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:04:30.578810 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:04:47.906050 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:04:47.906013 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 01:04:47.906628 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:04:47.906016 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 01:05:00.513081 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:05:00.513016 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:05:04.907170 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:05:04.907125 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" event={"ID":"5386e312-8c91-4461-b8b9-35b06185b5b8","Type":"ContainerStarted","Data":"2689f77edf85e1eb6a299df546a6107dac2bd8c0153a81f34e8d85a1e7decd6d"} Oct 31 01:05:04.907633 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:05:04.907198 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" event={"ID":"5386e312-8c91-4461-b8b9-35b06185b5b8","Type":"ContainerStarted","Data":"c31e1465bca0c36191ba25d73e8d67d696768bfc54d67bf77e736620b11b1eac"} Oct 31 01:05:04.907633 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:05:04.907241 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:05:04.907633 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:05:04.907282 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:05:04.936500 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:05:04.936448 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" podStartSLOduration=2.249825009 podStartE2EDuration="1m58.936432901s" totalImagesPullingTime="1m44.478469937s" totalInitContainerRuntime="12.208137955s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:04.934204386 +0000 UTC m=+2117.423479567" watchObservedRunningTime="2026-10-31 01:05:04.936432901 +0000 UTC m=+2117.425708080" Oct 31 01:05:10.916422 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:05:10.916389 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:05:30.508289 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:05:30.508245 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:06:00.509164 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:06:00.509127 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:06:30.509531 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:06:30.509492 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:06:41.680841 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:41.680808 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:06:52.689908 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:52.689859 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" podUID="5386e312-8c91-4461-b8b9-35b06185b5b8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:06:53.578730 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:53.578694 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245"] Oct 31 01:06:53.579014 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:53.578972 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" podUID="5386e312-8c91-4461-b8b9-35b06185b5b8" containerName="kserve-container" containerID="cri-o://c31e1465bca0c36191ba25d73e8d67d696768bfc54d67bf77e736620b11b1eac" gracePeriod=30 Oct 31 01:06:53.579122 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:53.579028 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" podUID="5386e312-8c91-4461-b8b9-35b06185b5b8" containerName="kube-rbac-proxy" containerID="cri-o://2689f77edf85e1eb6a299df546a6107dac2bd8c0153a81f34e8d85a1e7decd6d" gracePeriod=30 Oct 31 01:06:54.341273 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:54.341239 5939 generic.go:356] "Generic (PLEG): container finished" podID="5386e312-8c91-4461-b8b9-35b06185b5b8" containerID="2689f77edf85e1eb6a299df546a6107dac2bd8c0153a81f34e8d85a1e7decd6d" exitCode=0 Oct 31 01:06:54.341700 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:54.341309 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" event={"ID":"5386e312-8c91-4461-b8b9-35b06185b5b8","Type":"ContainerDied","Data":"2689f77edf85e1eb6a299df546a6107dac2bd8c0153a81f34e8d85a1e7decd6d"} Oct 31 01:06:55.226108 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.226080 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:06:55.265167 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.265100 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5386e312-8c91-4461-b8b9-35b06185b5b8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5386e312-8c91-4461-b8b9-35b06185b5b8-kube-rbac-proxy-sar-config\") pod \"5386e312-8c91-4461-b8b9-35b06185b5b8\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " Oct 31 01:06:55.265167 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.265143 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5386e312-8c91-4461-b8b9-35b06185b5b8-proxy-tls\" (UniqueName: \"kubernetes.io/secret/5386e312-8c91-4461-b8b9-35b06185b5b8-proxy-tls\") pod \"5386e312-8c91-4461-b8b9-35b06185b5b8\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " Oct 31 01:06:55.265376 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.265181 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5386e312-8c91-4461-b8b9-35b06185b5b8-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5386e312-8c91-4461-b8b9-35b06185b5b8-kserve-provision-location\") pod \"5386e312-8c91-4461-b8b9-35b06185b5b8\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " Oct 31 01:06:55.265376 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.265208 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5386e312-8c91-4461-b8b9-35b06185b5b8-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5386e312-8c91-4461-b8b9-35b06185b5b8-cabundle-cert\") pod \"5386e312-8c91-4461-b8b9-35b06185b5b8\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " Oct 31 01:06:55.265376 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.265256 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5386e312-8c91-4461-b8b9-35b06185b5b8-kube-api-access-kx4rm\" (UniqueName: \"kubernetes.io/projected/5386e312-8c91-4461-b8b9-35b06185b5b8-kube-api-access-kx4rm\") pod \"5386e312-8c91-4461-b8b9-35b06185b5b8\" (UID: \"5386e312-8c91-4461-b8b9-35b06185b5b8\") " Oct 31 01:06:55.265597 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.265577 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5386e312-8c91-4461-b8b9-35b06185b5b8-kube-rbac-proxy-sar-config" pod "5386e312-8c91-4461-b8b9-35b06185b5b8" (UID: "5386e312-8c91-4461-b8b9-35b06185b5b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:55.265690 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.265609 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5386e312-8c91-4461-b8b9-35b06185b5b8-cabundle-cert" pod "5386e312-8c91-4461-b8b9-35b06185b5b8" (UID: "5386e312-8c91-4461-b8b9-35b06185b5b8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:55.267857 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.267827 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5386e312-8c91-4461-b8b9-35b06185b5b8-kube-api-access-kx4rm" pod "5386e312-8c91-4461-b8b9-35b06185b5b8" (UID: "5386e312-8c91-4461-b8b9-35b06185b5b8"). InnerVolumeSpecName "kube-api-access-kx4rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:55.267985 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.267959 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5386e312-8c91-4461-b8b9-35b06185b5b8-proxy-tls" pod "5386e312-8c91-4461-b8b9-35b06185b5b8" (UID: "5386e312-8c91-4461-b8b9-35b06185b5b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:55.268972 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.268953 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5386e312-8c91-4461-b8b9-35b06185b5b8-kserve-provision-location" pod "5386e312-8c91-4461-b8b9-35b06185b5b8" (UID: "5386e312-8c91-4461-b8b9-35b06185b5b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:55.347449 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.347412 5939 generic.go:356] "Generic (PLEG): container finished" podID="5386e312-8c91-4461-b8b9-35b06185b5b8" containerID="c31e1465bca0c36191ba25d73e8d67d696768bfc54d67bf77e736620b11b1eac" exitCode=0 Oct 31 01:06:55.347953 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.347470 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" event={"ID":"5386e312-8c91-4461-b8b9-35b06185b5b8","Type":"ContainerDied","Data":"c31e1465bca0c36191ba25d73e8d67d696768bfc54d67bf77e736620b11b1eac"} Oct 31 01:06:55.347953 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.347518 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" event={"ID":"5386e312-8c91-4461-b8b9-35b06185b5b8","Type":"ContainerDied","Data":"d5a31f4f7d5b444ed8a6c6061298819e5514d53b18060d79f80c3872fdbfcbdb"} Oct 31 01:06:55.347953 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.347544 5939 scope.go:122] "RemoveContainer" containerID="2689f77edf85e1eb6a299df546a6107dac2bd8c0153a81f34e8d85a1e7decd6d" Oct 31 01:06:55.347953 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.347474 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245" Oct 31 01:06:55.356958 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.356942 5939 scope.go:122] "RemoveContainer" containerID="c31e1465bca0c36191ba25d73e8d67d696768bfc54d67bf77e736620b11b1eac" Oct 31 01:06:55.366907 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.366878 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx4rm\" (UniqueName: \"kubernetes.io/projected/5386e312-8c91-4461-b8b9-35b06185b5b8-kube-api-access-kx4rm\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:55.366907 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.366904 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5386e312-8c91-4461-b8b9-35b06185b5b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:55.367126 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.366920 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5386e312-8c91-4461-b8b9-35b06185b5b8-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:55.367126 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.366931 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5386e312-8c91-4461-b8b9-35b06185b5b8-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:55.367126 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.366940 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5386e312-8c91-4461-b8b9-35b06185b5b8-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:55.368576 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.368558 5939 scope.go:122] "RemoveContainer" containerID="19ab5ef695aac27e74fbf5a232349423406ff87a2abf4b54f9adf1edb75a3d0a" Oct 31 01:06:55.374108 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.374079 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245"] Oct 31 01:06:55.378445 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.378427 5939 scope.go:122] "RemoveContainer" containerID="2689f77edf85e1eb6a299df546a6107dac2bd8c0153a81f34e8d85a1e7decd6d" Oct 31 01:06:55.378764 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:06:55.378737 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2689f77edf85e1eb6a299df546a6107dac2bd8c0153a81f34e8d85a1e7decd6d\": container with ID starting with 2689f77edf85e1eb6a299df546a6107dac2bd8c0153a81f34e8d85a1e7decd6d not found: ID does not exist" containerID="2689f77edf85e1eb6a299df546a6107dac2bd8c0153a81f34e8d85a1e7decd6d" Oct 31 01:06:55.378833 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.378775 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2689f77edf85e1eb6a299df546a6107dac2bd8c0153a81f34e8d85a1e7decd6d"} err="failed to get container status \"2689f77edf85e1eb6a299df546a6107dac2bd8c0153a81f34e8d85a1e7decd6d\": rpc error: code = NotFound desc = could not find container \"2689f77edf85e1eb6a299df546a6107dac2bd8c0153a81f34e8d85a1e7decd6d\": container with ID starting with 2689f77edf85e1eb6a299df546a6107dac2bd8c0153a81f34e8d85a1e7decd6d not found: ID does not exist" Oct 31 01:06:55.378833 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.378794 5939 scope.go:122] "RemoveContainer" containerID="c31e1465bca0c36191ba25d73e8d67d696768bfc54d67bf77e736620b11b1eac" Oct 31 01:06:55.379103 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:06:55.379077 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c31e1465bca0c36191ba25d73e8d67d696768bfc54d67bf77e736620b11b1eac\": container with ID starting with c31e1465bca0c36191ba25d73e8d67d696768bfc54d67bf77e736620b11b1eac not found: ID does not exist" containerID="c31e1465bca0c36191ba25d73e8d67d696768bfc54d67bf77e736620b11b1eac" Oct 31 01:06:55.379139 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.379110 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c31e1465bca0c36191ba25d73e8d67d696768bfc54d67bf77e736620b11b1eac"} err="failed to get container status \"c31e1465bca0c36191ba25d73e8d67d696768bfc54d67bf77e736620b11b1eac\": rpc error: code = NotFound desc = could not find container \"c31e1465bca0c36191ba25d73e8d67d696768bfc54d67bf77e736620b11b1eac\": container with ID starting with c31e1465bca0c36191ba25d73e8d67d696768bfc54d67bf77e736620b11b1eac not found: ID does not exist" Oct 31 01:06:55.379139 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.379126 5939 scope.go:122] "RemoveContainer" containerID="19ab5ef695aac27e74fbf5a232349423406ff87a2abf4b54f9adf1edb75a3d0a" Oct 31 01:06:55.379400 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:06:55.379381 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ab5ef695aac27e74fbf5a232349423406ff87a2abf4b54f9adf1edb75a3d0a\": container with ID starting with 19ab5ef695aac27e74fbf5a232349423406ff87a2abf4b54f9adf1edb75a3d0a not found: ID does not exist" containerID="19ab5ef695aac27e74fbf5a232349423406ff87a2abf4b54f9adf1edb75a3d0a" Oct 31 01:06:55.379442 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.379407 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ab5ef695aac27e74fbf5a232349423406ff87a2abf4b54f9adf1edb75a3d0a"} err="failed to get container status \"19ab5ef695aac27e74fbf5a232349423406ff87a2abf4b54f9adf1edb75a3d0a\": rpc error: code = NotFound desc = could not find container \"19ab5ef695aac27e74fbf5a232349423406ff87a2abf4b54f9adf1edb75a3d0a\": container with ID starting with 19ab5ef695aac27e74fbf5a232349423406ff87a2abf4b54f9adf1edb75a3d0a not found: ID does not exist" Oct 31 01:06:55.382079 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.382056 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-autogluon-v2-variants-predictor-5694484bf-jj245"] Oct 31 01:06:55.651573 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:06:55.651539 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5386e312-8c91-4461-b8b9-35b06185b5b8" path="/var/lib/kubelet/pods/5386e312-8c91-4461-b8b9-35b06185b5b8/volumes" Oct 31 01:07:00.509967 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:07:00.509924 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:07:30.509347 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:07:30.509262 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:07:46.684616 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.684583 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs"] Oct 31 01:07:46.685101 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.684932 5939 cpu_manager.go:414] "containerMap: removing container" podUID="5386e312-8c91-4461-b8b9-35b06185b5b8" containerName="kserve-container" Oct 31 01:07:46.685101 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.684952 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5386e312-8c91-4461-b8b9-35b06185b5b8" containerName="kserve-container" Oct 31 01:07:46.685101 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.684976 5939 cpu_manager.go:414] "containerMap: removing container" podUID="5386e312-8c91-4461-b8b9-35b06185b5b8" containerName="storage-initializer" Oct 31 01:07:46.685101 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.684982 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5386e312-8c91-4461-b8b9-35b06185b5b8" containerName="storage-initializer" Oct 31 01:07:46.685101 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.685008 5939 cpu_manager.go:414] "containerMap: removing container" podUID="5386e312-8c91-4461-b8b9-35b06185b5b8" containerName="kube-rbac-proxy" Oct 31 01:07:46.685101 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.685013 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5386e312-8c91-4461-b8b9-35b06185b5b8" containerName="kube-rbac-proxy" Oct 31 01:07:46.685101 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.685065 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5386e312-8c91-4461-b8b9-35b06185b5b8" containerName="kserve-container" Oct 31 01:07:46.685101 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.685078 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5386e312-8c91-4461-b8b9-35b06185b5b8" containerName="kube-rbac-proxy" Oct 31 01:07:46.687089 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.687071 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:07:46.693122 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.693098 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 01:07:46.693997 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.693970 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 01:07:46.694087 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.694026 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 01:07:46.694087 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.694037 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\"" Oct 31 01:07:46.694087 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.694040 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 01:07:46.694194 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.694163 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 01:07:46.694254 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.694238 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-xgboost-predictor-serving-cert\"" Oct 31 01:07:46.702237 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.702208 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs"] Oct 31 01:07:46.709412 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.709390 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc6kt\" (UniqueName: \"kubernetes.io/projected/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kube-api-access-sc6kt\") pod \"isvc-xgboost-predictor-5795f6888f-8pqzs\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:07:46.709499 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.709430 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-5795f6888f-8pqzs\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:07:46.709499 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.709454 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-cabundle-cert\") pod \"isvc-xgboost-predictor-5795f6888f-8pqzs\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:07:46.709576 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.709553 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kserve-provision-location\") pod \"isvc-xgboost-predictor-5795f6888f-8pqzs\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:07:46.709691 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.709648 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-proxy-tls\") pod \"isvc-xgboost-predictor-5795f6888f-8pqzs\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:07:46.810933 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.810904 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-cabundle-cert\") pod \"isvc-xgboost-predictor-5795f6888f-8pqzs\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:07:46.811108 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.810938 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kserve-provision-location\") pod \"isvc-xgboost-predictor-5795f6888f-8pqzs\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:07:46.811108 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.810975 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-proxy-tls\") pod \"isvc-xgboost-predictor-5795f6888f-8pqzs\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:07:46.811108 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.811018 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6kt\" (UniqueName: \"kubernetes.io/projected/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kube-api-access-sc6kt\") pod \"isvc-xgboost-predictor-5795f6888f-8pqzs\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:07:46.811108 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.811050 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-5795f6888f-8pqzs\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:07:46.811418 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.811392 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kserve-provision-location\") pod \"isvc-xgboost-predictor-5795f6888f-8pqzs\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:07:46.811721 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.811698 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-cabundle-cert\") pod \"isvc-xgboost-predictor-5795f6888f-8pqzs\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:07:46.811798 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.811752 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-5795f6888f-8pqzs\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:07:46.813399 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.813381 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-proxy-tls\") pod \"isvc-xgboost-predictor-5795f6888f-8pqzs\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:07:46.825752 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:46.825728 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc6kt\" (UniqueName: \"kubernetes.io/projected/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kube-api-access-sc6kt\") pod \"isvc-xgboost-predictor-5795f6888f-8pqzs\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:07:47.000987 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:47.000906 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:07:47.281511 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:47.281426 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs"] Oct 31 01:07:47.285250 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:07:47.285222 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8b4499_a0b8_4557_9775_c6bb3dc989d3.slice/crio-1aa583f96844f1dde29afe9e57b27291ab25fd7036690e6c62554fa1b40cd44c WatchSource:0}: Error finding container 1aa583f96844f1dde29afe9e57b27291ab25fd7036690e6c62554fa1b40cd44c: Status 404 returned error can't find the container with id 1aa583f96844f1dde29afe9e57b27291ab25fd7036690e6c62554fa1b40cd44c Oct 31 01:07:47.552579 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:47.552497 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" event={"ID":"4d8b4499-a0b8-4557-9775-c6bb3dc989d3","Type":"ContainerStarted","Data":"71702e929d3a958410f1cada9f993d6757eed5dbd5079d7a50d6a5d282e87ab8"} Oct 31 01:07:47.552579 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:47.552560 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" event={"ID":"4d8b4499-a0b8-4557-9775-c6bb3dc989d3","Type":"ContainerStarted","Data":"1aa583f96844f1dde29afe9e57b27291ab25fd7036690e6c62554fa1b40cd44c"} Oct 31 01:07:51.569973 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:51.569883 5939 generic.go:356] "Generic (PLEG): container finished" podID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerID="71702e929d3a958410f1cada9f993d6757eed5dbd5079d7a50d6a5d282e87ab8" exitCode=0 Oct 31 01:07:51.569973 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:07:51.569953 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" event={"ID":"4d8b4499-a0b8-4557-9775-c6bb3dc989d3","Type":"ContainerDied","Data":"71702e929d3a958410f1cada9f993d6757eed5dbd5079d7a50d6a5d282e87ab8"} Oct 31 01:08:00.513299 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:08:00.513254 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:08:09.670676 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:08:09.670637 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" event={"ID":"4d8b4499-a0b8-4557-9775-c6bb3dc989d3","Type":"ContainerStarted","Data":"f556f9a96576f741318c0cba8c3edc94ac58025a69228a668122804265693fc4"} Oct 31 01:08:09.671078 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:08:09.670723 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" event={"ID":"4d8b4499-a0b8-4557-9775-c6bb3dc989d3","Type":"ContainerStarted","Data":"e385cf419eb74beb581604615d079c4db060ec0568d3ab2414c0c33546227512"} Oct 31 01:08:09.671078 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:08:09.670974 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:08:09.699261 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:08:09.699207 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" podStartSLOduration=2.184861629 podStartE2EDuration="23.699193447s" totalImagesPullingTime="17.648441687s" totalInitContainerRuntime="3.865890131s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:09.698451833 +0000 UTC m=+2302.187727011" watchObservedRunningTime="2026-10-31 01:08:09.699193447 +0000 UTC m=+2302.188468625" Oct 31 01:08:10.675115 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:08:10.675071 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:08:10.676287 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:08:10.676246 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Oct 31 01:08:11.678392 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:08:11.678351 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Oct 31 01:08:16.682803 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:08:16.682772 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:08:16.683270 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:08:16.683245 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Oct 31 01:08:26.683646 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:08:26.683601 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Oct 31 01:08:30.509997 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:08:30.509961 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:08:36.684084 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:08:36.684043 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Oct 31 01:08:46.683369 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:08:46.683323 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Oct 31 01:08:56.683868 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:08:56.683828 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Oct 31 01:09:00.509095 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:09:00.509017 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:09:06.683341 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:06.683303 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Oct 31 01:09:16.683300 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:16.683263 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Oct 31 01:09:26.684616 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:26.684586 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:09:30.509180 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:09:30.509142 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:09:37.503908 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:37.503874 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs"] Oct 31 01:09:37.506199 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:37.504117 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kserve-container" containerID="cri-o://e385cf419eb74beb581604615d079c4db060ec0568d3ab2414c0c33546227512" gracePeriod=30 Oct 31 01:09:37.506199 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:37.504151 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kube-rbac-proxy" containerID="cri-o://f556f9a96576f741318c0cba8c3edc94ac58025a69228a668122804265693fc4" gracePeriod=30 Oct 31 01:09:38.036083 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:38.036050 5939 generic.go:356] "Generic (PLEG): container finished" podID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerID="f556f9a96576f741318c0cba8c3edc94ac58025a69228a668122804265693fc4" exitCode=0 Oct 31 01:09:38.036264 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:38.036117 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" event={"ID":"4d8b4499-a0b8-4557-9775-c6bb3dc989d3","Type":"ContainerDied","Data":"f556f9a96576f741318c0cba8c3edc94ac58025a69228a668122804265693fc4"} Oct 31 01:09:41.679589 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:41.679541 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused" Oct 31 01:09:41.845316 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:41.845291 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:09:41.915735 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:41.915627 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-proxy-tls\") pod \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " Oct 31 01:09:41.915886 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:41.915773 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kube-api-access-sc6kt\" (UniqueName: \"kubernetes.io/projected/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kube-api-access-sc6kt\") pod \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " Oct 31 01:09:41.915886 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:41.915807 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kube-rbac-proxy-sar-config\") pod \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " Oct 31 01:09:41.915886 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:41.915851 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-cabundle-cert\") pod \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " Oct 31 01:09:41.916013 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:41.915989 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kserve-provision-location\") pod \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\" (UID: \"4d8b4499-a0b8-4557-9775-c6bb3dc989d3\") " Oct 31 01:09:41.916188 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:41.916163 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-cabundle-cert" pod "4d8b4499-a0b8-4557-9775-c6bb3dc989d3" (UID: "4d8b4499-a0b8-4557-9775-c6bb3dc989d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:41.916324 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:41.916180 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kube-rbac-proxy-sar-config" pod "4d8b4499-a0b8-4557-9775-c6bb3dc989d3" (UID: "4d8b4499-a0b8-4557-9775-c6bb3dc989d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:41.916377 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:41.916312 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kserve-provision-location" pod "4d8b4499-a0b8-4557-9775-c6bb3dc989d3" (UID: "4d8b4499-a0b8-4557-9775-c6bb3dc989d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:41.916377 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:41.916320 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:41.917942 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:41.917921 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-proxy-tls" pod "4d8b4499-a0b8-4557-9775-c6bb3dc989d3" (UID: "4d8b4499-a0b8-4557-9775-c6bb3dc989d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:41.918046 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:41.917959 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kube-api-access-sc6kt" pod "4d8b4499-a0b8-4557-9775-c6bb3dc989d3" (UID: "4d8b4499-a0b8-4557-9775-c6bb3dc989d3"). InnerVolumeSpecName "kube-api-access-sc6kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:42.017001 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.016971 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc6kt\" (UniqueName: \"kubernetes.io/projected/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kube-api-access-sc6kt\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:42.017001 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.016999 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:42.017001 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.017009 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:42.017246 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.017019 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d8b4499-a0b8-4557-9775-c6bb3dc989d3-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:42.054217 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.054187 5939 generic.go:356] "Generic (PLEG): container finished" podID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerID="e385cf419eb74beb581604615d079c4db060ec0568d3ab2414c0c33546227512" exitCode=0 Oct 31 01:09:42.054360 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.054237 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" event={"ID":"4d8b4499-a0b8-4557-9775-c6bb3dc989d3","Type":"ContainerDied","Data":"e385cf419eb74beb581604615d079c4db060ec0568d3ab2414c0c33546227512"} Oct 31 01:09:42.054360 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.054250 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" Oct 31 01:09:42.054360 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.054304 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs" event={"ID":"4d8b4499-a0b8-4557-9775-c6bb3dc989d3","Type":"ContainerDied","Data":"1aa583f96844f1dde29afe9e57b27291ab25fd7036690e6c62554fa1b40cd44c"} Oct 31 01:09:42.054360 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.054332 5939 scope.go:122] "RemoveContainer" containerID="f556f9a96576f741318c0cba8c3edc94ac58025a69228a668122804265693fc4" Oct 31 01:09:42.063529 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.063509 5939 scope.go:122] "RemoveContainer" containerID="e385cf419eb74beb581604615d079c4db060ec0568d3ab2414c0c33546227512" Oct 31 01:09:42.070862 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.070847 5939 scope.go:122] "RemoveContainer" containerID="71702e929d3a958410f1cada9f993d6757eed5dbd5079d7a50d6a5d282e87ab8" Oct 31 01:09:42.078203 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.078188 5939 scope.go:122] "RemoveContainer" containerID="f556f9a96576f741318c0cba8c3edc94ac58025a69228a668122804265693fc4" Oct 31 01:09:42.078457 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:09:42.078437 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f556f9a96576f741318c0cba8c3edc94ac58025a69228a668122804265693fc4\": container with ID starting with f556f9a96576f741318c0cba8c3edc94ac58025a69228a668122804265693fc4 not found: ID does not exist" containerID="f556f9a96576f741318c0cba8c3edc94ac58025a69228a668122804265693fc4" Oct 31 01:09:42.078495 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.078468 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f556f9a96576f741318c0cba8c3edc94ac58025a69228a668122804265693fc4"} err="failed to get container status \"f556f9a96576f741318c0cba8c3edc94ac58025a69228a668122804265693fc4\": rpc error: code = NotFound desc = could not find container \"f556f9a96576f741318c0cba8c3edc94ac58025a69228a668122804265693fc4\": container with ID starting with f556f9a96576f741318c0cba8c3edc94ac58025a69228a668122804265693fc4 not found: ID does not exist" Oct 31 01:09:42.078495 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.078486 5939 scope.go:122] "RemoveContainer" containerID="e385cf419eb74beb581604615d079c4db060ec0568d3ab2414c0c33546227512" Oct 31 01:09:42.078756 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:09:42.078725 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e385cf419eb74beb581604615d079c4db060ec0568d3ab2414c0c33546227512\": container with ID starting with e385cf419eb74beb581604615d079c4db060ec0568d3ab2414c0c33546227512 not found: ID does not exist" containerID="e385cf419eb74beb581604615d079c4db060ec0568d3ab2414c0c33546227512" Oct 31 01:09:42.078836 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.078758 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e385cf419eb74beb581604615d079c4db060ec0568d3ab2414c0c33546227512"} err="failed to get container status \"e385cf419eb74beb581604615d079c4db060ec0568d3ab2414c0c33546227512\": rpc error: code = NotFound desc = could not find container \"e385cf419eb74beb581604615d079c4db060ec0568d3ab2414c0c33546227512\": container with ID starting with e385cf419eb74beb581604615d079c4db060ec0568d3ab2414c0c33546227512 not found: ID does not exist" Oct 31 01:09:42.078836 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.078778 5939 scope.go:122] "RemoveContainer" containerID="71702e929d3a958410f1cada9f993d6757eed5dbd5079d7a50d6a5d282e87ab8" Oct 31 01:09:42.079133 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:09:42.079112 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71702e929d3a958410f1cada9f993d6757eed5dbd5079d7a50d6a5d282e87ab8\": container with ID starting with 71702e929d3a958410f1cada9f993d6757eed5dbd5079d7a50d6a5d282e87ab8 not found: ID does not exist" containerID="71702e929d3a958410f1cada9f993d6757eed5dbd5079d7a50d6a5d282e87ab8" Oct 31 01:09:42.079229 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.079137 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71702e929d3a958410f1cada9f993d6757eed5dbd5079d7a50d6a5d282e87ab8"} err="failed to get container status \"71702e929d3a958410f1cada9f993d6757eed5dbd5079d7a50d6a5d282e87ab8\": rpc error: code = NotFound desc = could not find container \"71702e929d3a958410f1cada9f993d6757eed5dbd5079d7a50d6a5d282e87ab8\": container with ID starting with 71702e929d3a958410f1cada9f993d6757eed5dbd5079d7a50d6a5d282e87ab8 not found: ID does not exist" Oct 31 01:09:42.080809 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.080789 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs"] Oct 31 01:09:42.086614 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.086594 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-predictor-5795f6888f-8pqzs"] Oct 31 01:09:42.972289 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.972248 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t"] Oct 31 01:09:42.972714 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.972573 5939 cpu_manager.go:414] "containerMap: removing container" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kube-rbac-proxy" Oct 31 01:09:42.972714 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.972584 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kube-rbac-proxy" Oct 31 01:09:42.972714 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.972599 5939 cpu_manager.go:414] "containerMap: removing container" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kserve-container" Oct 31 01:09:42.972714 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.972604 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kserve-container" Oct 31 01:09:42.972714 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.972613 5939 cpu_manager.go:414] "containerMap: removing container" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="storage-initializer" Oct 31 01:09:42.972714 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.972619 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="storage-initializer" Oct 31 01:09:42.972714 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.972701 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kube-rbac-proxy" Oct 31 01:09:42.972714 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.972711 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" containerName="kserve-container" Oct 31 01:09:42.977158 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.977137 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:42.983206 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.983185 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 01:09:42.983351 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.983222 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 01:09:42.983351 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.983224 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\"" Oct 31 01:09:42.983351 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.983264 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 01:09:42.983952 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.983933 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 01:09:42.984045 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.983939 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 01:09:42.984045 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.984026 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\"" Oct 31 01:09:42.989738 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:42.989718 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t"] Oct 31 01:09:43.025955 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.025929 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/902a8870-1397-43e9-98ab-26e74cbb6aad-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:43.026105 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.025967 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z9mf\" (UniqueName: \"kubernetes.io/projected/902a8870-1397-43e9-98ab-26e74cbb6aad-kube-api-access-8z9mf\") pod \"isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:43.026105 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.025990 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902a8870-1397-43e9-98ab-26e74cbb6aad-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:43.026178 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.026121 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902a8870-1397-43e9-98ab-26e74cbb6aad-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:43.026220 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.026174 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902a8870-1397-43e9-98ab-26e74cbb6aad-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:43.127186 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.127151 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902a8870-1397-43e9-98ab-26e74cbb6aad-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:43.127363 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.127203 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902a8870-1397-43e9-98ab-26e74cbb6aad-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:43.127363 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.127228 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/902a8870-1397-43e9-98ab-26e74cbb6aad-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:43.127363 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.127255 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9mf\" (UniqueName: \"kubernetes.io/projected/902a8870-1397-43e9-98ab-26e74cbb6aad-kube-api-access-8z9mf\") pod \"isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:43.127363 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.127291 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902a8870-1397-43e9-98ab-26e74cbb6aad-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:43.127642 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.127621 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902a8870-1397-43e9-98ab-26e74cbb6aad-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:43.128090 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.128068 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902a8870-1397-43e9-98ab-26e74cbb6aad-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:43.128132 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.128068 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/902a8870-1397-43e9-98ab-26e74cbb6aad-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:43.129653 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.129630 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902a8870-1397-43e9-98ab-26e74cbb6aad-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:43.141895 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.141869 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z9mf\" (UniqueName: \"kubernetes.io/projected/902a8870-1397-43e9-98ab-26e74cbb6aad-kube-api-access-8z9mf\") pod \"isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:43.294355 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.294273 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:43.575313 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.575225 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t"] Oct 31 01:09:43.578379 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:09:43.578352 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902a8870_1397_43e9_98ab_26e74cbb6aad.slice/crio-c6d875d89299b8950d5cf4ba98a58f066f3435b4e65a4140f8032ecc0520fd21 WatchSource:0}: Error finding container c6d875d89299b8950d5cf4ba98a58f066f3435b4e65a4140f8032ecc0520fd21: Status 404 returned error can't find the container with id c6d875d89299b8950d5cf4ba98a58f066f3435b4e65a4140f8032ecc0520fd21 Oct 31 01:09:43.657317 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:43.657280 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d8b4499-a0b8-4557-9775-c6bb3dc989d3" path="/var/lib/kubelet/pods/4d8b4499-a0b8-4557-9775-c6bb3dc989d3/volumes" Oct 31 01:09:44.064750 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:44.064715 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" event={"ID":"902a8870-1397-43e9-98ab-26e74cbb6aad","Type":"ContainerStarted","Data":"6ae27fabbdb43a8d319f256613dbc6db2dc5bc0c82ce0af96ac1f121d1fce75d"} Oct 31 01:09:44.065159 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:44.064779 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" event={"ID":"902a8870-1397-43e9-98ab-26e74cbb6aad","Type":"ContainerStarted","Data":"c6d875d89299b8950d5cf4ba98a58f066f3435b4e65a4140f8032ecc0520fd21"} Oct 31 01:09:47.933940 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:47.933914 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 01:09:47.934915 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:47.934894 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 01:09:48.084032 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:48.084002 5939 generic.go:356] "Generic (PLEG): container finished" podID="902a8870-1397-43e9-98ab-26e74cbb6aad" containerID="6ae27fabbdb43a8d319f256613dbc6db2dc5bc0c82ce0af96ac1f121d1fce75d" exitCode=0 Oct 31 01:09:48.084219 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:48.084060 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" event={"ID":"902a8870-1397-43e9-98ab-26e74cbb6aad","Type":"ContainerDied","Data":"6ae27fabbdb43a8d319f256613dbc6db2dc5bc0c82ce0af96ac1f121d1fce75d"} Oct 31 01:09:49.090388 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:49.090350 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" event={"ID":"902a8870-1397-43e9-98ab-26e74cbb6aad","Type":"ContainerStarted","Data":"6d67a3930c10b8d0ba49a7ecb92e4984609a20b4b4f7c88953fd3b1d94e199c3"} Oct 31 01:09:49.090819 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:49.090412 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" event={"ID":"902a8870-1397-43e9-98ab-26e74cbb6aad","Type":"ContainerStarted","Data":"d6a88f5e84e5250bce311aa4236e07e8534d2e73e70b2b64f2939be3c7c21afc"} Oct 31 01:09:49.090819 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:49.090601 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:49.090819 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:49.090651 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:09:49.121239 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:49.121194 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" podStartSLOduration=3.213919505 podStartE2EDuration="7.121180226s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.907260721s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:49.119018523 +0000 UTC m=+2401.608293702" watchObservedRunningTime="2026-10-31 01:09:49.121180226 +0000 UTC m=+2401.610455406" Oct 31 01:09:55.100091 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:09:55.100065 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:10:00.509059 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:10:00.509021 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:10:25.104341 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:25.104308 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:10:30.508636 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:10:30.508556 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:10:33.245255 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:33.245226 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t"] Oct 31 01:10:33.245720 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:33.245471 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" podUID="902a8870-1397-43e9-98ab-26e74cbb6aad" containerName="kserve-container" containerID="cri-o://d6a88f5e84e5250bce311aa4236e07e8534d2e73e70b2b64f2939be3c7c21afc" gracePeriod=30 Oct 31 01:10:33.245720 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:33.245537 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" podUID="902a8870-1397-43e9-98ab-26e74cbb6aad" containerName="kube-rbac-proxy" containerID="cri-o://6d67a3930c10b8d0ba49a7ecb92e4984609a20b4b4f7c88953fd3b1d94e199c3" gracePeriod=30 Oct 31 01:10:34.275302 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:34.275262 5939 generic.go:356] "Generic (PLEG): container finished" podID="902a8870-1397-43e9-98ab-26e74cbb6aad" containerID="6d67a3930c10b8d0ba49a7ecb92e4984609a20b4b4f7c88953fd3b1d94e199c3" exitCode=0 Oct 31 01:10:34.275694 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:34.275337 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" event={"ID":"902a8870-1397-43e9-98ab-26e74cbb6aad","Type":"ContainerDied","Data":"6d67a3930c10b8d0ba49a7ecb92e4984609a20b4b4f7c88953fd3b1d94e199c3"} Oct 31 01:10:35.094844 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:35.094808 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" podUID="902a8870-1397-43e9-98ab-26e74cbb6aad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused" Oct 31 01:10:39.896207 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:39.896173 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:10:40.026188 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.026104 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902a8870-1397-43e9-98ab-26e74cbb6aad-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/902a8870-1397-43e9-98ab-26e74cbb6aad-cabundle-cert\") pod \"902a8870-1397-43e9-98ab-26e74cbb6aad\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " Oct 31 01:10:40.026188 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.026171 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902a8870-1397-43e9-98ab-26e74cbb6aad-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902a8870-1397-43e9-98ab-26e74cbb6aad-kserve-provision-location\") pod \"902a8870-1397-43e9-98ab-26e74cbb6aad\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " Oct 31 01:10:40.026383 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.026199 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902a8870-1397-43e9-98ab-26e74cbb6aad-kube-api-access-8z9mf\" (UniqueName: \"kubernetes.io/projected/902a8870-1397-43e9-98ab-26e74cbb6aad-kube-api-access-8z9mf\") pod \"902a8870-1397-43e9-98ab-26e74cbb6aad\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " Oct 31 01:10:40.026383 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.026265 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902a8870-1397-43e9-98ab-26e74cbb6aad-proxy-tls\" (UniqueName: \"kubernetes.io/secret/902a8870-1397-43e9-98ab-26e74cbb6aad-proxy-tls\") pod \"902a8870-1397-43e9-98ab-26e74cbb6aad\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " Oct 31 01:10:40.026383 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.026336 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902a8870-1397-43e9-98ab-26e74cbb6aad-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902a8870-1397-43e9-98ab-26e74cbb6aad-kube-rbac-proxy-sar-config\") pod \"902a8870-1397-43e9-98ab-26e74cbb6aad\" (UID: \"902a8870-1397-43e9-98ab-26e74cbb6aad\") " Oct 31 01:10:40.026543 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.026511 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902a8870-1397-43e9-98ab-26e74cbb6aad-kserve-provision-location" pod "902a8870-1397-43e9-98ab-26e74cbb6aad" (UID: "902a8870-1397-43e9-98ab-26e74cbb6aad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:10:40.026543 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.026525 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902a8870-1397-43e9-98ab-26e74cbb6aad-cabundle-cert" pod "902a8870-1397-43e9-98ab-26e74cbb6aad" (UID: "902a8870-1397-43e9-98ab-26e74cbb6aad"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:10:40.026776 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.026756 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/902a8870-1397-43e9-98ab-26e74cbb6aad-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:40.026853 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.026782 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902a8870-1397-43e9-98ab-26e74cbb6aad-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:40.026853 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.026784 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902a8870-1397-43e9-98ab-26e74cbb6aad-kube-rbac-proxy-sar-config" pod "902a8870-1397-43e9-98ab-26e74cbb6aad" (UID: "902a8870-1397-43e9-98ab-26e74cbb6aad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:10:40.028412 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.028392 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902a8870-1397-43e9-98ab-26e74cbb6aad-proxy-tls" pod "902a8870-1397-43e9-98ab-26e74cbb6aad" (UID: "902a8870-1397-43e9-98ab-26e74cbb6aad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:10:40.028481 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.028442 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902a8870-1397-43e9-98ab-26e74cbb6aad-kube-api-access-8z9mf" pod "902a8870-1397-43e9-98ab-26e74cbb6aad" (UID: "902a8870-1397-43e9-98ab-26e74cbb6aad"). InnerVolumeSpecName "kube-api-access-8z9mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:10:40.128177 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.128135 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902a8870-1397-43e9-98ab-26e74cbb6aad-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:40.128177 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.128169 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902a8870-1397-43e9-98ab-26e74cbb6aad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:40.128177 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.128181 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z9mf\" (UniqueName: \"kubernetes.io/projected/902a8870-1397-43e9-98ab-26e74cbb6aad-kube-api-access-8z9mf\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:40.298913 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.298823 5939 generic.go:356] "Generic (PLEG): container finished" podID="902a8870-1397-43e9-98ab-26e74cbb6aad" containerID="d6a88f5e84e5250bce311aa4236e07e8534d2e73e70b2b64f2939be3c7c21afc" exitCode=0 Oct 31 01:10:40.298913 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.298883 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" Oct 31 01:10:40.299180 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.298907 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" event={"ID":"902a8870-1397-43e9-98ab-26e74cbb6aad","Type":"ContainerDied","Data":"d6a88f5e84e5250bce311aa4236e07e8534d2e73e70b2b64f2939be3c7c21afc"} Oct 31 01:10:40.299180 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.298968 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t" event={"ID":"902a8870-1397-43e9-98ab-26e74cbb6aad","Type":"ContainerDied","Data":"c6d875d89299b8950d5cf4ba98a58f066f3435b4e65a4140f8032ecc0520fd21"} Oct 31 01:10:40.299180 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.298995 5939 scope.go:122] "RemoveContainer" containerID="6d67a3930c10b8d0ba49a7ecb92e4984609a20b4b4f7c88953fd3b1d94e199c3" Oct 31 01:10:40.307987 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.307964 5939 scope.go:122] "RemoveContainer" containerID="d6a88f5e84e5250bce311aa4236e07e8534d2e73e70b2b64f2939be3c7c21afc" Oct 31 01:10:40.315580 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.315563 5939 scope.go:122] "RemoveContainer" containerID="6ae27fabbdb43a8d319f256613dbc6db2dc5bc0c82ce0af96ac1f121d1fce75d" Oct 31 01:10:40.322968 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.322951 5939 scope.go:122] "RemoveContainer" containerID="6d67a3930c10b8d0ba49a7ecb92e4984609a20b4b4f7c88953fd3b1d94e199c3" Oct 31 01:10:40.323234 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:10:40.323216 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d67a3930c10b8d0ba49a7ecb92e4984609a20b4b4f7c88953fd3b1d94e199c3\": container with ID starting with 6d67a3930c10b8d0ba49a7ecb92e4984609a20b4b4f7c88953fd3b1d94e199c3 not found: ID does not exist" containerID="6d67a3930c10b8d0ba49a7ecb92e4984609a20b4b4f7c88953fd3b1d94e199c3" Oct 31 01:10:40.323284 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.323244 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d67a3930c10b8d0ba49a7ecb92e4984609a20b4b4f7c88953fd3b1d94e199c3"} err="failed to get container status \"6d67a3930c10b8d0ba49a7ecb92e4984609a20b4b4f7c88953fd3b1d94e199c3\": rpc error: code = NotFound desc = could not find container \"6d67a3930c10b8d0ba49a7ecb92e4984609a20b4b4f7c88953fd3b1d94e199c3\": container with ID starting with 6d67a3930c10b8d0ba49a7ecb92e4984609a20b4b4f7c88953fd3b1d94e199c3 not found: ID does not exist" Oct 31 01:10:40.323284 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.323260 5939 scope.go:122] "RemoveContainer" containerID="d6a88f5e84e5250bce311aa4236e07e8534d2e73e70b2b64f2939be3c7c21afc" Oct 31 01:10:40.323507 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:10:40.323488 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a88f5e84e5250bce311aa4236e07e8534d2e73e70b2b64f2939be3c7c21afc\": container with ID starting with d6a88f5e84e5250bce311aa4236e07e8534d2e73e70b2b64f2939be3c7c21afc not found: ID does not exist" containerID="d6a88f5e84e5250bce311aa4236e07e8534d2e73e70b2b64f2939be3c7c21afc" Oct 31 01:10:40.323548 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.323515 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a88f5e84e5250bce311aa4236e07e8534d2e73e70b2b64f2939be3c7c21afc"} err="failed to get container status \"d6a88f5e84e5250bce311aa4236e07e8534d2e73e70b2b64f2939be3c7c21afc\": rpc error: code = NotFound desc = could not find container \"d6a88f5e84e5250bce311aa4236e07e8534d2e73e70b2b64f2939be3c7c21afc\": container with ID starting with d6a88f5e84e5250bce311aa4236e07e8534d2e73e70b2b64f2939be3c7c21afc not found: ID does not exist" Oct 31 01:10:40.323548 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.323531 5939 scope.go:122] "RemoveContainer" containerID="6ae27fabbdb43a8d319f256613dbc6db2dc5bc0c82ce0af96ac1f121d1fce75d" Oct 31 01:10:40.323839 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:10:40.323778 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ae27fabbdb43a8d319f256613dbc6db2dc5bc0c82ce0af96ac1f121d1fce75d\": container with ID starting with 6ae27fabbdb43a8d319f256613dbc6db2dc5bc0c82ce0af96ac1f121d1fce75d not found: ID does not exist" containerID="6ae27fabbdb43a8d319f256613dbc6db2dc5bc0c82ce0af96ac1f121d1fce75d" Oct 31 01:10:40.323839 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.323801 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae27fabbdb43a8d319f256613dbc6db2dc5bc0c82ce0af96ac1f121d1fce75d"} err="failed to get container status \"6ae27fabbdb43a8d319f256613dbc6db2dc5bc0c82ce0af96ac1f121d1fce75d\": rpc error: code = NotFound desc = could not find container \"6ae27fabbdb43a8d319f256613dbc6db2dc5bc0c82ce0af96ac1f121d1fce75d\": container with ID starting with 6ae27fabbdb43a8d319f256613dbc6db2dc5bc0c82ce0af96ac1f121d1fce75d not found: ID does not exist" Oct 31 01:10:40.325634 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.325615 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t"] Oct 31 01:10:40.335116 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.335095 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-mlserver-predictor-84594cbb55-2js4t"] Oct 31 01:10:40.727294 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.727259 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk"] Oct 31 01:10:40.727590 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.727578 5939 cpu_manager.go:414] "containerMap: removing container" podUID="902a8870-1397-43e9-98ab-26e74cbb6aad" containerName="kserve-container" Oct 31 01:10:40.727691 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.727591 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902a8870-1397-43e9-98ab-26e74cbb6aad" containerName="kserve-container" Oct 31 01:10:40.727691 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.727623 5939 cpu_manager.go:414] "containerMap: removing container" podUID="902a8870-1397-43e9-98ab-26e74cbb6aad" containerName="storage-initializer" Oct 31 01:10:40.727691 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.727629 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902a8870-1397-43e9-98ab-26e74cbb6aad" containerName="storage-initializer" Oct 31 01:10:40.727691 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.727641 5939 cpu_manager.go:414] "containerMap: removing container" podUID="902a8870-1397-43e9-98ab-26e74cbb6aad" containerName="kube-rbac-proxy" Oct 31 01:10:40.727691 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.727648 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902a8870-1397-43e9-98ab-26e74cbb6aad" containerName="kube-rbac-proxy" Oct 31 01:10:40.727844 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.727705 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="902a8870-1397-43e9-98ab-26e74cbb6aad" containerName="kserve-container" Oct 31 01:10:40.727844 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.727719 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="902a8870-1397-43e9-98ab-26e74cbb6aad" containerName="kube-rbac-proxy" Oct 31 01:10:40.732366 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.732345 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:40.745338 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.745311 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 01:10:40.745526 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.745422 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\"" Oct 31 01:10:40.745526 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.745511 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 01:10:40.745714 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.745530 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 01:10:40.745714 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.745566 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 01:10:40.745714 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.745614 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"xgboost-v2-mlserver-predictor-serving-cert\"" Oct 31 01:10:40.745714 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.745623 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 01:10:40.747183 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.747160 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk"] Oct 31 01:10:40.834890 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.834858 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b1964b9-0445-4de4-8404-495044b7c020-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:40.834890 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.834894 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b1964b9-0445-4de4-8404-495044b7c020-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:40.835112 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.834913 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b1964b9-0445-4de4-8404-495044b7c020-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:40.835150 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.835102 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1964b9-0445-4de4-8404-495044b7c020-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:40.835150 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.835134 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8dlt\" (UniqueName: \"kubernetes.io/projected/0b1964b9-0445-4de4-8404-495044b7c020-kube-api-access-c8dlt\") pod \"xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:40.936359 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.936315 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b1964b9-0445-4de4-8404-495044b7c020-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:40.936359 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.936361 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b1964b9-0445-4de4-8404-495044b7c020-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:40.936953 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.936378 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b1964b9-0445-4de4-8404-495044b7c020-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:40.936953 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.936451 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1964b9-0445-4de4-8404-495044b7c020-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:40.936953 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.936476 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8dlt\" (UniqueName: \"kubernetes.io/projected/0b1964b9-0445-4de4-8404-495044b7c020-kube-api-access-c8dlt\") pod \"xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:40.936953 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.936945 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1964b9-0445-4de4-8404-495044b7c020-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:40.937192 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.937132 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b1964b9-0445-4de4-8404-495044b7c020-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:40.937247 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.937197 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b1964b9-0445-4de4-8404-495044b7c020-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:40.938860 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.938841 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b1964b9-0445-4de4-8404-495044b7c020-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:40.952092 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:40.952061 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8dlt\" (UniqueName: \"kubernetes.io/projected/0b1964b9-0445-4de4-8404-495044b7c020-kube-api-access-c8dlt\") pod \"xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:41.053900 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:41.053814 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:41.335803 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:41.335726 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk"] Oct 31 01:10:41.339006 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:10:41.338978 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1964b9_0445_4de4_8404_495044b7c020.slice/crio-cd166d3fee5288997906c1830c2253284c9cdb6f46ff941a87933a9d0b08b2ff WatchSource:0}: Error finding container cd166d3fee5288997906c1830c2253284c9cdb6f46ff941a87933a9d0b08b2ff: Status 404 returned error can't find the container with id cd166d3fee5288997906c1830c2253284c9cdb6f46ff941a87933a9d0b08b2ff Oct 31 01:10:41.652868 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:41.652838 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="902a8870-1397-43e9-98ab-26e74cbb6aad" path="/var/lib/kubelet/pods/902a8870-1397-43e9-98ab-26e74cbb6aad/volumes" Oct 31 01:10:42.310626 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:42.310591 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" event={"ID":"0b1964b9-0445-4de4-8404-495044b7c020","Type":"ContainerStarted","Data":"c152c2d457ba9f8bb4140f89f26b503513b7ccc95608fcd59f8c1a636b5fd70f"} Oct 31 01:10:42.311034 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:42.310650 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" event={"ID":"0b1964b9-0445-4de4-8404-495044b7c020","Type":"ContainerStarted","Data":"cd166d3fee5288997906c1830c2253284c9cdb6f46ff941a87933a9d0b08b2ff"} Oct 31 01:10:46.031740 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.031704 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh"] Oct 31 01:10:46.035391 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.035367 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:46.041737 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.041678 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-create-v2-predictor-serving-cert\"" Oct 31 01:10:46.041737 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.041726 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\"" Oct 31 01:10:46.044379 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.044350 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\"" Oct 31 01:10:46.045822 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.045802 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\"" Oct 31 01:10:46.045974 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.045960 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 01:10:46.046094 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.046072 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\"" Oct 31 01:10:46.050683 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.050651 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh"] Oct 31 01:10:46.092956 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.092926 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr"] Oct 31 01:10:46.097695 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.097222 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:46.110855 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.110830 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr"] Oct 31 01:10:46.111237 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.111220 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\"" Oct 31 01:10:46.111237 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.111231 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-create-predictor-serving-cert\"" Oct 31 01:10:46.182082 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.182052 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9040ce87-14fd-4f52-acc2-3368cda11568-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5d6fb757bb-khqxh\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:46.182237 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.182086 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9040ce87-14fd-4f52-acc2-3368cda11568-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5d6fb757bb-khqxh\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:46.182237 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.182109 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cf9f156-749c-4050-a0e7-1efabf7550a2-cabundle-cert\") pod \"isvc-canary-create-predictor-74fddf7499-hfnvr\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:46.182237 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.182214 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cf9f156-749c-4050-a0e7-1efabf7550a2-proxy-tls\") pod \"isvc-canary-create-predictor-74fddf7499-hfnvr\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:46.182380 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.182242 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cf9f156-749c-4050-a0e7-1efabf7550a2-kserve-provision-location\") pod \"isvc-canary-create-predictor-74fddf7499-hfnvr\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:46.182380 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.182345 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9040ce87-14fd-4f52-acc2-3368cda11568-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5d6fb757bb-khqxh\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:46.182380 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.182366 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtljx\" (UniqueName: \"kubernetes.io/projected/9040ce87-14fd-4f52-acc2-3368cda11568-kube-api-access-rtljx\") pod \"isvc-canary-create-v2-predictor-5d6fb757bb-khqxh\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:46.182503 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.182384 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjwr8\" (UniqueName: \"kubernetes.io/projected/4cf9f156-749c-4050-a0e7-1efabf7550a2-kube-api-access-jjwr8\") pod \"isvc-canary-create-predictor-74fddf7499-hfnvr\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:46.182503 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.182405 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cf9f156-749c-4050-a0e7-1efabf7550a2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-74fddf7499-hfnvr\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:46.182503 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.182471 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9040ce87-14fd-4f52-acc2-3368cda11568-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5d6fb757bb-khqxh\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:46.283454 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.283369 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9040ce87-14fd-4f52-acc2-3368cda11568-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5d6fb757bb-khqxh\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:46.283454 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.283404 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtljx\" (UniqueName: \"kubernetes.io/projected/9040ce87-14fd-4f52-acc2-3368cda11568-kube-api-access-rtljx\") pod \"isvc-canary-create-v2-predictor-5d6fb757bb-khqxh\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:46.283454 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.283423 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjwr8\" (UniqueName: \"kubernetes.io/projected/4cf9f156-749c-4050-a0e7-1efabf7550a2-kube-api-access-jjwr8\") pod \"isvc-canary-create-predictor-74fddf7499-hfnvr\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:46.283454 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.283444 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cf9f156-749c-4050-a0e7-1efabf7550a2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-74fddf7499-hfnvr\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:46.283835 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.283560 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9040ce87-14fd-4f52-acc2-3368cda11568-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5d6fb757bb-khqxh\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:46.283835 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.283611 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9040ce87-14fd-4f52-acc2-3368cda11568-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5d6fb757bb-khqxh\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:46.283835 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.283643 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9040ce87-14fd-4f52-acc2-3368cda11568-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5d6fb757bb-khqxh\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:46.283835 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.283695 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cf9f156-749c-4050-a0e7-1efabf7550a2-cabundle-cert\") pod \"isvc-canary-create-predictor-74fddf7499-hfnvr\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:46.283835 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.283743 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cf9f156-749c-4050-a0e7-1efabf7550a2-proxy-tls\") pod \"isvc-canary-create-predictor-74fddf7499-hfnvr\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:46.283835 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.283770 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cf9f156-749c-4050-a0e7-1efabf7550a2-kserve-provision-location\") pod \"isvc-canary-create-predictor-74fddf7499-hfnvr\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:46.284090 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.284068 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9040ce87-14fd-4f52-acc2-3368cda11568-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5d6fb757bb-khqxh\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:46.284140 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.284124 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cf9f156-749c-4050-a0e7-1efabf7550a2-kserve-provision-location\") pod \"isvc-canary-create-predictor-74fddf7499-hfnvr\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:46.284296 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.284264 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cf9f156-749c-4050-a0e7-1efabf7550a2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-74fddf7499-hfnvr\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:46.284406 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.284383 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9040ce87-14fd-4f52-acc2-3368cda11568-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5d6fb757bb-khqxh\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:46.284459 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.284413 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cf9f156-749c-4050-a0e7-1efabf7550a2-cabundle-cert\") pod \"isvc-canary-create-predictor-74fddf7499-hfnvr\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:46.284772 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.284751 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9040ce87-14fd-4f52-acc2-3368cda11568-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5d6fb757bb-khqxh\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:46.286234 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.286203 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9040ce87-14fd-4f52-acc2-3368cda11568-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5d6fb757bb-khqxh\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:46.286333 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.286283 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cf9f156-749c-4050-a0e7-1efabf7550a2-proxy-tls\") pod \"isvc-canary-create-predictor-74fddf7499-hfnvr\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:46.307108 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.307086 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjwr8\" (UniqueName: \"kubernetes.io/projected/4cf9f156-749c-4050-a0e7-1efabf7550a2-kube-api-access-jjwr8\") pod \"isvc-canary-create-predictor-74fddf7499-hfnvr\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:46.307215 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.307108 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtljx\" (UniqueName: \"kubernetes.io/projected/9040ce87-14fd-4f52-acc2-3368cda11568-kube-api-access-rtljx\") pod \"isvc-canary-create-v2-predictor-5d6fb757bb-khqxh\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:46.329071 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.329047 5939 generic.go:356] "Generic (PLEG): container finished" podID="0b1964b9-0445-4de4-8404-495044b7c020" containerID="c152c2d457ba9f8bb4140f89f26b503513b7ccc95608fcd59f8c1a636b5fd70f" exitCode=0 Oct 31 01:10:46.329186 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.329094 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" event={"ID":"0b1964b9-0445-4de4-8404-495044b7c020","Type":"ContainerDied","Data":"c152c2d457ba9f8bb4140f89f26b503513b7ccc95608fcd59f8c1a636b5fd70f"} Oct 31 01:10:46.353643 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.353620 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:46.416997 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.416760 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:46.640473 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.640372 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh"] Oct 31 01:10:46.643134 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:10:46.643099 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9040ce87_14fd_4f52_acc2_3368cda11568.slice/crio-b8f48596fa46017cdfa2f7a84e4350f45801349f37d6e0d771c3cd9f0d4db2da WatchSource:0}: Error finding container b8f48596fa46017cdfa2f7a84e4350f45801349f37d6e0d771c3cd9f0d4db2da: Status 404 returned error can't find the container with id b8f48596fa46017cdfa2f7a84e4350f45801349f37d6e0d771c3cd9f0d4db2da Oct 31 01:10:46.706224 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:46.706201 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr"] Oct 31 01:10:46.709194 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:10:46.709171 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf9f156_749c_4050_a0e7_1efabf7550a2.slice/crio-b759cf53516952e4e4da15fb05d990f3565b218ebc3d19b73e162825dee43fad WatchSource:0}: Error finding container b759cf53516952e4e4da15fb05d990f3565b218ebc3d19b73e162825dee43fad: Status 404 returned error can't find the container with id b759cf53516952e4e4da15fb05d990f3565b218ebc3d19b73e162825dee43fad Oct 31 01:10:47.335657 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:47.335614 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" event={"ID":"4cf9f156-749c-4050-a0e7-1efabf7550a2","Type":"ContainerStarted","Data":"ed386a70ca64ca6d1dc195325931f8dbc373264174c6aebd7695cbe3adbee9e1"} Oct 31 01:10:47.336160 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:47.335722 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" event={"ID":"4cf9f156-749c-4050-a0e7-1efabf7550a2","Type":"ContainerStarted","Data":"b759cf53516952e4e4da15fb05d990f3565b218ebc3d19b73e162825dee43fad"} Oct 31 01:10:47.337423 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:47.337398 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" event={"ID":"9040ce87-14fd-4f52-acc2-3368cda11568","Type":"ContainerStarted","Data":"2e6d81bd4e789b679f9630d46bba182b87b9dafe485417bcc4af51813340bcd0"} Oct 31 01:10:47.337559 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:47.337456 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" event={"ID":"9040ce87-14fd-4f52-acc2-3368cda11568","Type":"ContainerStarted","Data":"b8f48596fa46017cdfa2f7a84e4350f45801349f37d6e0d771c3cd9f0d4db2da"} Oct 31 01:10:47.339851 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:47.339829 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" event={"ID":"0b1964b9-0445-4de4-8404-495044b7c020","Type":"ContainerStarted","Data":"e014caa2f72e12a3d024d6ba8c3ffdc63589b56a1b4e9a240cda0f78ff00aed8"} Oct 31 01:10:47.339963 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:47.339876 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" event={"ID":"0b1964b9-0445-4de4-8404-495044b7c020","Type":"ContainerStarted","Data":"92c6305f4a22a0286d027aa592c013b30ad059724c998dc66763e73e7490f8ff"} Oct 31 01:10:47.340081 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:47.340064 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:47.340126 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:47.340112 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:47.386989 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:47.386930 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" podStartSLOduration=3.367684197 podStartE2EDuration="7.386916359s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.019232162s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:47.38638973 +0000 UTC m=+2459.875664910" watchObservedRunningTime="2026-10-31 01:10:47.386916359 +0000 UTC m=+2459.876191538" Oct 31 01:10:50.354984 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:50.354955 5939 generic.go:356] "Generic (PLEG): container finished" podID="9040ce87-14fd-4f52-acc2-3368cda11568" containerID="2e6d81bd4e789b679f9630d46bba182b87b9dafe485417bcc4af51813340bcd0" exitCode=0 Oct 31 01:10:50.355320 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:50.355015 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" event={"ID":"9040ce87-14fd-4f52-acc2-3368cda11568","Type":"ContainerDied","Data":"2e6d81bd4e789b679f9630d46bba182b87b9dafe485417bcc4af51813340bcd0"} Oct 31 01:10:51.360328 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:51.360282 5939 generic.go:356] "Generic (PLEG): container finished" podID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerID="ed386a70ca64ca6d1dc195325931f8dbc373264174c6aebd7695cbe3adbee9e1" exitCode=0 Oct 31 01:10:51.360791 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:51.360346 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" event={"ID":"4cf9f156-749c-4050-a0e7-1efabf7550a2","Type":"ContainerDied","Data":"ed386a70ca64ca6d1dc195325931f8dbc373264174c6aebd7695cbe3adbee9e1"} Oct 31 01:10:51.362678 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:51.362644 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" event={"ID":"9040ce87-14fd-4f52-acc2-3368cda11568","Type":"ContainerStarted","Data":"8fc2f49dcf94e9963162d32b5e405c258983c3c44440cc02b93b7c5456fdf579"} Oct 31 01:10:51.362777 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:51.362725 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" event={"ID":"9040ce87-14fd-4f52-acc2-3368cda11568","Type":"ContainerStarted","Data":"bcd79fe72993e6b7fc3e89d1e37677e3bc215d1b6704bf84c05bd32ed07a3183"} Oct 31 01:10:51.363207 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:51.363188 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:51.363306 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:51.363233 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:51.364514 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:51.364487 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused" Oct 31 01:10:51.413652 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:51.413592 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" podStartSLOduration=1.776526679 podStartE2EDuration="5.413574326s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.637047647s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:51.41284099 +0000 UTC m=+2463.902116170" watchObservedRunningTime="2026-10-31 01:10:51.413574326 +0000 UTC m=+2463.902849502" Oct 31 01:10:52.369327 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:52.369290 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" event={"ID":"4cf9f156-749c-4050-a0e7-1efabf7550a2","Type":"ContainerStarted","Data":"2b74712ee1b4aa52a2c7db1eb2f6b436b915d12488cad142bca01ea62fdba912"} Oct 31 01:10:52.369760 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:52.369359 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" event={"ID":"4cf9f156-749c-4050-a0e7-1efabf7550a2","Type":"ContainerStarted","Data":"f63675ff1952730adec7dafc840a5a4a5575991edfc780ce030bb85777aad71a"} Oct 31 01:10:52.369760 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:52.369551 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused" Oct 31 01:10:52.369914 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:52.369791 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:52.369914 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:52.369844 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:52.371020 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:52.370999 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused" Oct 31 01:10:52.398613 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:52.398561 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" podStartSLOduration=2.386829214 podStartE2EDuration="6.398549241s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.011720027s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:52.397749614 +0000 UTC m=+2464.887024793" watchObservedRunningTime="2026-10-31 01:10:52.398549241 +0000 UTC m=+2464.887824420" Oct 31 01:10:53.349635 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:53.349607 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:10:53.374094 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:53.374053 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused" Oct 31 01:10:57.374349 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:57.374314 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:10:57.375011 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:57.374983 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused" Oct 31 01:10:58.378128 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:58.378102 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:10:58.378631 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:10:58.378607 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused" Oct 31 01:11:00.508800 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:11:00.508759 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:11:07.375400 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:07.375362 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused" Oct 31 01:11:08.378879 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:08.378835 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused" Oct 31 01:11:17.375822 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:17.375773 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused" Oct 31 01:11:18.378902 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:18.378861 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused" Oct 31 01:11:23.353552 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:23.353522 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:11:27.375107 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:27.375060 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused" Oct 31 01:11:28.379248 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:28.379211 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused" Oct 31 01:11:30.509699 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:11:30.509645 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:11:31.052216 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:31.052186 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk"] Oct 31 01:11:31.052438 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:31.052415 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" podUID="0b1964b9-0445-4de4-8404-495044b7c020" containerName="kserve-container" containerID="cri-o://92c6305f4a22a0286d027aa592c013b30ad059724c998dc66763e73e7490f8ff" gracePeriod=30 Oct 31 01:11:31.052505 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:31.052465 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" podUID="0b1964b9-0445-4de4-8404-495044b7c020" containerName="kube-rbac-proxy" containerID="cri-o://e014caa2f72e12a3d024d6ba8c3ffdc63589b56a1b4e9a240cda0f78ff00aed8" gracePeriod=30 Oct 31 01:11:31.544980 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:31.544948 5939 generic.go:356] "Generic (PLEG): container finished" podID="0b1964b9-0445-4de4-8404-495044b7c020" containerID="e014caa2f72e12a3d024d6ba8c3ffdc63589b56a1b4e9a240cda0f78ff00aed8" exitCode=0 Oct 31 01:11:31.545374 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:31.545029 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" event={"ID":"0b1964b9-0445-4de4-8404-495044b7c020","Type":"ContainerDied","Data":"e014caa2f72e12a3d024d6ba8c3ffdc63589b56a1b4e9a240cda0f78ff00aed8"} Oct 31 01:11:33.345531 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:33.345489 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" podUID="0b1964b9-0445-4de4-8404-495044b7c020" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused" Oct 31 01:11:37.375452 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:37.375411 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused" Oct 31 01:11:37.897626 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:37.897596 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:11:38.045575 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.045484 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b1964b9-0445-4de4-8404-495044b7c020-proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b1964b9-0445-4de4-8404-495044b7c020-proxy-tls\") pod \"0b1964b9-0445-4de4-8404-495044b7c020\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " Oct 31 01:11:38.045575 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.045534 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b1964b9-0445-4de4-8404-495044b7c020-kube-api-access-c8dlt\" (UniqueName: \"kubernetes.io/projected/0b1964b9-0445-4de4-8404-495044b7c020-kube-api-access-c8dlt\") pod \"0b1964b9-0445-4de4-8404-495044b7c020\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " Oct 31 01:11:38.045853 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.045637 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b1964b9-0445-4de4-8404-495044b7c020-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b1964b9-0445-4de4-8404-495044b7c020-cabundle-cert\") pod \"0b1964b9-0445-4de4-8404-495044b7c020\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " Oct 31 01:11:38.045853 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.045706 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b1964b9-0445-4de4-8404-495044b7c020-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b1964b9-0445-4de4-8404-495044b7c020-kube-rbac-proxy-sar-config\") pod \"0b1964b9-0445-4de4-8404-495044b7c020\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " Oct 31 01:11:38.045853 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.045763 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b1964b9-0445-4de4-8404-495044b7c020-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1964b9-0445-4de4-8404-495044b7c020-kserve-provision-location\") pod \"0b1964b9-0445-4de4-8404-495044b7c020\" (UID: \"0b1964b9-0445-4de4-8404-495044b7c020\") " Oct 31 01:11:38.046114 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.046083 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1964b9-0445-4de4-8404-495044b7c020-kube-rbac-proxy-sar-config" pod "0b1964b9-0445-4de4-8404-495044b7c020" (UID: "0b1964b9-0445-4de4-8404-495044b7c020"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:38.046114 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.046022 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1964b9-0445-4de4-8404-495044b7c020-cabundle-cert" pod "0b1964b9-0445-4de4-8404-495044b7c020" (UID: "0b1964b9-0445-4de4-8404-495044b7c020"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:38.046282 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.046136 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1964b9-0445-4de4-8404-495044b7c020-kserve-provision-location" pod "0b1964b9-0445-4de4-8404-495044b7c020" (UID: "0b1964b9-0445-4de4-8404-495044b7c020"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:38.047694 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.047658 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1964b9-0445-4de4-8404-495044b7c020-proxy-tls" pod "0b1964b9-0445-4de4-8404-495044b7c020" (UID: "0b1964b9-0445-4de4-8404-495044b7c020"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:38.047694 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.047661 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1964b9-0445-4de4-8404-495044b7c020-kube-api-access-c8dlt" pod "0b1964b9-0445-4de4-8404-495044b7c020" (UID: "0b1964b9-0445-4de4-8404-495044b7c020"). InnerVolumeSpecName "kube-api-access-c8dlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:38.147342 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.147310 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b1964b9-0445-4de4-8404-495044b7c020-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:38.147342 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.147337 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8dlt\" (UniqueName: \"kubernetes.io/projected/0b1964b9-0445-4de4-8404-495044b7c020-kube-api-access-c8dlt\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:38.147342 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.147347 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b1964b9-0445-4de4-8404-495044b7c020-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:38.147569 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.147356 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b1964b9-0445-4de4-8404-495044b7c020-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:38.147569 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.147367 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1964b9-0445-4de4-8404-495044b7c020-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:38.378848 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.378809 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused" Oct 31 01:11:38.576986 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.576950 5939 generic.go:356] "Generic (PLEG): container finished" podID="0b1964b9-0445-4de4-8404-495044b7c020" containerID="92c6305f4a22a0286d027aa592c013b30ad059724c998dc66763e73e7490f8ff" exitCode=0 Oct 31 01:11:38.577147 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.577017 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" Oct 31 01:11:38.577147 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.577035 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" event={"ID":"0b1964b9-0445-4de4-8404-495044b7c020","Type":"ContainerDied","Data":"92c6305f4a22a0286d027aa592c013b30ad059724c998dc66763e73e7490f8ff"} Oct 31 01:11:38.577147 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.577097 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk" event={"ID":"0b1964b9-0445-4de4-8404-495044b7c020","Type":"ContainerDied","Data":"cd166d3fee5288997906c1830c2253284c9cdb6f46ff941a87933a9d0b08b2ff"} Oct 31 01:11:38.577147 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.577124 5939 scope.go:122] "RemoveContainer" containerID="e014caa2f72e12a3d024d6ba8c3ffdc63589b56a1b4e9a240cda0f78ff00aed8" Oct 31 01:11:38.586295 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.586274 5939 scope.go:122] "RemoveContainer" containerID="92c6305f4a22a0286d027aa592c013b30ad059724c998dc66763e73e7490f8ff" Oct 31 01:11:38.595637 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.595612 5939 scope.go:122] "RemoveContainer" containerID="c152c2d457ba9f8bb4140f89f26b503513b7ccc95608fcd59f8c1a636b5fd70f" Oct 31 01:11:38.604979 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.604945 5939 scope.go:122] "RemoveContainer" containerID="e014caa2f72e12a3d024d6ba8c3ffdc63589b56a1b4e9a240cda0f78ff00aed8" Oct 31 01:11:38.605285 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:11:38.605263 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e014caa2f72e12a3d024d6ba8c3ffdc63589b56a1b4e9a240cda0f78ff00aed8\": container with ID starting with e014caa2f72e12a3d024d6ba8c3ffdc63589b56a1b4e9a240cda0f78ff00aed8 not found: ID does not exist" containerID="e014caa2f72e12a3d024d6ba8c3ffdc63589b56a1b4e9a240cda0f78ff00aed8" Oct 31 01:11:38.605368 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.605292 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e014caa2f72e12a3d024d6ba8c3ffdc63589b56a1b4e9a240cda0f78ff00aed8"} err="failed to get container status \"e014caa2f72e12a3d024d6ba8c3ffdc63589b56a1b4e9a240cda0f78ff00aed8\": rpc error: code = NotFound desc = could not find container \"e014caa2f72e12a3d024d6ba8c3ffdc63589b56a1b4e9a240cda0f78ff00aed8\": container with ID starting with e014caa2f72e12a3d024d6ba8c3ffdc63589b56a1b4e9a240cda0f78ff00aed8 not found: ID does not exist" Oct 31 01:11:38.605368 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.605308 5939 scope.go:122] "RemoveContainer" containerID="92c6305f4a22a0286d027aa592c013b30ad059724c998dc66763e73e7490f8ff" Oct 31 01:11:38.605572 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:11:38.605555 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c6305f4a22a0286d027aa592c013b30ad059724c998dc66763e73e7490f8ff\": container with ID starting with 92c6305f4a22a0286d027aa592c013b30ad059724c998dc66763e73e7490f8ff not found: ID does not exist" containerID="92c6305f4a22a0286d027aa592c013b30ad059724c998dc66763e73e7490f8ff" Oct 31 01:11:38.605637 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.605582 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c6305f4a22a0286d027aa592c013b30ad059724c998dc66763e73e7490f8ff"} err="failed to get container status \"92c6305f4a22a0286d027aa592c013b30ad059724c998dc66763e73e7490f8ff\": rpc error: code = NotFound desc = could not find container \"92c6305f4a22a0286d027aa592c013b30ad059724c998dc66763e73e7490f8ff\": container with ID starting with 92c6305f4a22a0286d027aa592c013b30ad059724c998dc66763e73e7490f8ff not found: ID does not exist" Oct 31 01:11:38.605637 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.605604 5939 scope.go:122] "RemoveContainer" containerID="c152c2d457ba9f8bb4140f89f26b503513b7ccc95608fcd59f8c1a636b5fd70f" Oct 31 01:11:38.605881 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:11:38.605859 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c152c2d457ba9f8bb4140f89f26b503513b7ccc95608fcd59f8c1a636b5fd70f\": container with ID starting with c152c2d457ba9f8bb4140f89f26b503513b7ccc95608fcd59f8c1a636b5fd70f not found: ID does not exist" containerID="c152c2d457ba9f8bb4140f89f26b503513b7ccc95608fcd59f8c1a636b5fd70f" Oct 31 01:11:38.605931 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.605890 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c152c2d457ba9f8bb4140f89f26b503513b7ccc95608fcd59f8c1a636b5fd70f"} err="failed to get container status \"c152c2d457ba9f8bb4140f89f26b503513b7ccc95608fcd59f8c1a636b5fd70f\": rpc error: code = NotFound desc = could not find container \"c152c2d457ba9f8bb4140f89f26b503513b7ccc95608fcd59f8c1a636b5fd70f\": container with ID starting with c152c2d457ba9f8bb4140f89f26b503513b7ccc95608fcd59f8c1a636b5fd70f not found: ID does not exist" Oct 31 01:11:38.605967 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.605927 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk"] Oct 31 01:11:38.614869 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:38.614846 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/xgboost-v2-mlserver-predictor-57f66dbc9d-2k4hk"] Oct 31 01:11:39.652127 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:39.652085 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b1964b9-0445-4de4-8404-495044b7c020" path="/var/lib/kubelet/pods/0b1964b9-0445-4de4-8404-495044b7c020/volumes" Oct 31 01:11:47.375680 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:47.375630 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused" Oct 31 01:11:48.379302 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:48.379264 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused" Oct 31 01:11:57.375767 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:57.375726 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused" Oct 31 01:11:58.379062 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:58.378980 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused" Oct 31 01:11:58.648121 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:11:58.648018 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused" Oct 31 01:12:00.509708 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:12:00.509655 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:12:07.375540 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:07.375493 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused" Oct 31 01:12:08.647958 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:08.647919 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused" Oct 31 01:12:16.648297 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:16.648268 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:12:16.862949 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:16.862915 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr"] Oct 31 01:12:16.863198 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:16.863165 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kserve-container" containerID="cri-o://f63675ff1952730adec7dafc840a5a4a5575991edfc780ce030bb85777aad71a" gracePeriod=30 Oct 31 01:12:16.863351 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:16.863237 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kube-rbac-proxy" containerID="cri-o://2b74712ee1b4aa52a2c7db1eb2f6b436b915d12488cad142bca01ea62fdba912" gracePeriod=30 Oct 31 01:12:16.864026 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:16.863840 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:12:17.746735 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:17.746700 5939 generic.go:356] "Generic (PLEG): container finished" podID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerID="2b74712ee1b4aa52a2c7db1eb2f6b436b915d12488cad142bca01ea62fdba912" exitCode=0 Oct 31 01:12:17.747128 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:17.746774 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" event={"ID":"4cf9f156-749c-4050-a0e7-1efabf7550a2","Type":"ContainerDied","Data":"2b74712ee1b4aa52a2c7db1eb2f6b436b915d12488cad142bca01ea62fdba912"} Oct 31 01:12:18.374368 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:18.374326 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused" Oct 31 01:12:22.002481 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.002453 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:12:22.133060 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.133032 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cf9f156-749c-4050-a0e7-1efabf7550a2-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cf9f156-749c-4050-a0e7-1efabf7550a2-cabundle-cert\") pod \"4cf9f156-749c-4050-a0e7-1efabf7550a2\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " Oct 31 01:12:22.133236 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.133095 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf9f156-749c-4050-a0e7-1efabf7550a2-kube-api-access-jjwr8\" (UniqueName: \"kubernetes.io/projected/4cf9f156-749c-4050-a0e7-1efabf7550a2-kube-api-access-jjwr8\") pod \"4cf9f156-749c-4050-a0e7-1efabf7550a2\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " Oct 31 01:12:22.133236 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.133218 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf9f156-749c-4050-a0e7-1efabf7550a2-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cf9f156-749c-4050-a0e7-1efabf7550a2-kserve-provision-location\") pod \"4cf9f156-749c-4050-a0e7-1efabf7550a2\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " Oct 31 01:12:22.133323 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.133260 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cf9f156-749c-4050-a0e7-1efabf7550a2-proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cf9f156-749c-4050-a0e7-1efabf7550a2-proxy-tls\") pod \"4cf9f156-749c-4050-a0e7-1efabf7550a2\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " Oct 31 01:12:22.133323 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.133292 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cf9f156-749c-4050-a0e7-1efabf7550a2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cf9f156-749c-4050-a0e7-1efabf7550a2-kube-rbac-proxy-sar-config\") pod \"4cf9f156-749c-4050-a0e7-1efabf7550a2\" (UID: \"4cf9f156-749c-4050-a0e7-1efabf7550a2\") " Oct 31 01:12:22.133580 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.133540 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf9f156-749c-4050-a0e7-1efabf7550a2-cabundle-cert" pod "4cf9f156-749c-4050-a0e7-1efabf7550a2" (UID: "4cf9f156-749c-4050-a0e7-1efabf7550a2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:12:22.133580 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.133556 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf9f156-749c-4050-a0e7-1efabf7550a2-kserve-provision-location" pod "4cf9f156-749c-4050-a0e7-1efabf7550a2" (UID: "4cf9f156-749c-4050-a0e7-1efabf7550a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:22.133750 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.133649 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf9f156-749c-4050-a0e7-1efabf7550a2-kube-rbac-proxy-sar-config" pod "4cf9f156-749c-4050-a0e7-1efabf7550a2" (UID: "4cf9f156-749c-4050-a0e7-1efabf7550a2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:12:22.135151 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.135127 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf9f156-749c-4050-a0e7-1efabf7550a2-proxy-tls" pod "4cf9f156-749c-4050-a0e7-1efabf7550a2" (UID: "4cf9f156-749c-4050-a0e7-1efabf7550a2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:12:22.135253 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.135237 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf9f156-749c-4050-a0e7-1efabf7550a2-kube-api-access-jjwr8" pod "4cf9f156-749c-4050-a0e7-1efabf7550a2" (UID: "4cf9f156-749c-4050-a0e7-1efabf7550a2"). InnerVolumeSpecName "kube-api-access-jjwr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:12:22.234411 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.234376 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjwr8\" (UniqueName: \"kubernetes.io/projected/4cf9f156-749c-4050-a0e7-1efabf7550a2-kube-api-access-jjwr8\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:12:22.234411 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.234405 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cf9f156-749c-4050-a0e7-1efabf7550a2-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:12:22.234411 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.234415 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cf9f156-749c-4050-a0e7-1efabf7550a2-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:12:22.234643 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.234423 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cf9f156-749c-4050-a0e7-1efabf7550a2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:12:22.234643 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.234433 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cf9f156-749c-4050-a0e7-1efabf7550a2-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:12:22.770489 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.770452 5939 generic.go:356] "Generic (PLEG): container finished" podID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerID="f63675ff1952730adec7dafc840a5a4a5575991edfc780ce030bb85777aad71a" exitCode=0 Oct 31 01:12:22.770726 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.770516 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" event={"ID":"4cf9f156-749c-4050-a0e7-1efabf7550a2","Type":"ContainerDied","Data":"f63675ff1952730adec7dafc840a5a4a5575991edfc780ce030bb85777aad71a"} Oct 31 01:12:22.770726 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.770534 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" Oct 31 01:12:22.770726 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.770564 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr" event={"ID":"4cf9f156-749c-4050-a0e7-1efabf7550a2","Type":"ContainerDied","Data":"b759cf53516952e4e4da15fb05d990f3565b218ebc3d19b73e162825dee43fad"} Oct 31 01:12:22.770726 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.770590 5939 scope.go:122] "RemoveContainer" containerID="2b74712ee1b4aa52a2c7db1eb2f6b436b915d12488cad142bca01ea62fdba912" Oct 31 01:12:22.779511 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.779489 5939 scope.go:122] "RemoveContainer" containerID="f63675ff1952730adec7dafc840a5a4a5575991edfc780ce030bb85777aad71a" Oct 31 01:12:22.787422 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.787400 5939 scope.go:122] "RemoveContainer" containerID="ed386a70ca64ca6d1dc195325931f8dbc373264174c6aebd7695cbe3adbee9e1" Oct 31 01:12:22.794476 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.794452 5939 scope.go:122] "RemoveContainer" containerID="2b74712ee1b4aa52a2c7db1eb2f6b436b915d12488cad142bca01ea62fdba912" Oct 31 01:12:22.795296 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:12:22.794784 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b74712ee1b4aa52a2c7db1eb2f6b436b915d12488cad142bca01ea62fdba912\": container with ID starting with 2b74712ee1b4aa52a2c7db1eb2f6b436b915d12488cad142bca01ea62fdba912 not found: ID does not exist" containerID="2b74712ee1b4aa52a2c7db1eb2f6b436b915d12488cad142bca01ea62fdba912" Oct 31 01:12:22.795296 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.794852 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b74712ee1b4aa52a2c7db1eb2f6b436b915d12488cad142bca01ea62fdba912"} err="failed to get container status \"2b74712ee1b4aa52a2c7db1eb2f6b436b915d12488cad142bca01ea62fdba912\": rpc error: code = NotFound desc = could not find container \"2b74712ee1b4aa52a2c7db1eb2f6b436b915d12488cad142bca01ea62fdba912\": container with ID starting with 2b74712ee1b4aa52a2c7db1eb2f6b436b915d12488cad142bca01ea62fdba912 not found: ID does not exist" Oct 31 01:12:22.795296 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.794877 5939 scope.go:122] "RemoveContainer" containerID="f63675ff1952730adec7dafc840a5a4a5575991edfc780ce030bb85777aad71a" Oct 31 01:12:22.795296 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:12:22.795258 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63675ff1952730adec7dafc840a5a4a5575991edfc780ce030bb85777aad71a\": container with ID starting with f63675ff1952730adec7dafc840a5a4a5575991edfc780ce030bb85777aad71a not found: ID does not exist" containerID="f63675ff1952730adec7dafc840a5a4a5575991edfc780ce030bb85777aad71a" Oct 31 01:12:22.795296 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.795281 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63675ff1952730adec7dafc840a5a4a5575991edfc780ce030bb85777aad71a"} err="failed to get container status \"f63675ff1952730adec7dafc840a5a4a5575991edfc780ce030bb85777aad71a\": rpc error: code = NotFound desc = could not find container \"f63675ff1952730adec7dafc840a5a4a5575991edfc780ce030bb85777aad71a\": container with ID starting with f63675ff1952730adec7dafc840a5a4a5575991edfc780ce030bb85777aad71a not found: ID does not exist" Oct 31 01:12:22.795296 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.795298 5939 scope.go:122] "RemoveContainer" containerID="ed386a70ca64ca6d1dc195325931f8dbc373264174c6aebd7695cbe3adbee9e1" Oct 31 01:12:22.795653 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:12:22.795592 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed386a70ca64ca6d1dc195325931f8dbc373264174c6aebd7695cbe3adbee9e1\": container with ID starting with ed386a70ca64ca6d1dc195325931f8dbc373264174c6aebd7695cbe3adbee9e1 not found: ID does not exist" containerID="ed386a70ca64ca6d1dc195325931f8dbc373264174c6aebd7695cbe3adbee9e1" Oct 31 01:12:22.795653 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.795614 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed386a70ca64ca6d1dc195325931f8dbc373264174c6aebd7695cbe3adbee9e1"} err="failed to get container status \"ed386a70ca64ca6d1dc195325931f8dbc373264174c6aebd7695cbe3adbee9e1\": rpc error: code = NotFound desc = could not find container \"ed386a70ca64ca6d1dc195325931f8dbc373264174c6aebd7695cbe3adbee9e1\": container with ID starting with ed386a70ca64ca6d1dc195325931f8dbc373264174c6aebd7695cbe3adbee9e1 not found: ID does not exist" Oct 31 01:12:22.796260 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.796242 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr"] Oct 31 01:12:22.804885 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:22.804861 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-predictor-74fddf7499-hfnvr"] Oct 31 01:12:23.652155 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:23.652122 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" path="/var/lib/kubelet/pods/4cf9f156-749c-4050-a0e7-1efabf7550a2/volumes" Oct 31 01:12:30.511360 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:12:30.511326 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:12:36.362436 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:36.362400 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh"] Oct 31 01:12:36.362968 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:36.362636 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kserve-container" containerID="cri-o://bcd79fe72993e6b7fc3e89d1e37677e3bc215d1b6704bf84c05bd32ed07a3183" gracePeriod=30 Oct 31 01:12:36.362968 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:36.362652 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kube-rbac-proxy" containerID="cri-o://8fc2f49dcf94e9963162d32b5e405c258983c3c44440cc02b93b7c5456fdf579" gracePeriod=30 Oct 31 01:12:36.648011 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:36.647919 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused" Oct 31 01:12:36.833793 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:36.833763 5939 generic.go:356] "Generic (PLEG): container finished" podID="9040ce87-14fd-4f52-acc2-3368cda11568" containerID="8fc2f49dcf94e9963162d32b5e405c258983c3c44440cc02b93b7c5456fdf579" exitCode=0 Oct 31 01:12:36.833964 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:36.833832 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" event={"ID":"9040ce87-14fd-4f52-acc2-3368cda11568","Type":"ContainerDied","Data":"8fc2f49dcf94e9963162d32b5e405c258983c3c44440cc02b93b7c5456fdf579"} Oct 31 01:12:37.370798 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:37.370750 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused" Oct 31 01:12:41.303822 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.303794 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:12:41.393234 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.393150 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9040ce87-14fd-4f52-acc2-3368cda11568-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9040ce87-14fd-4f52-acc2-3368cda11568-proxy-tls\") pod \"9040ce87-14fd-4f52-acc2-3368cda11568\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " Oct 31 01:12:41.393234 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.393194 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9040ce87-14fd-4f52-acc2-3368cda11568-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9040ce87-14fd-4f52-acc2-3368cda11568-cabundle-cert\") pod \"9040ce87-14fd-4f52-acc2-3368cda11568\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " Oct 31 01:12:41.393234 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.393222 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9040ce87-14fd-4f52-acc2-3368cda11568-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9040ce87-14fd-4f52-acc2-3368cda11568-kserve-provision-location\") pod \"9040ce87-14fd-4f52-acc2-3368cda11568\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " Oct 31 01:12:41.393500 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.393308 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9040ce87-14fd-4f52-acc2-3368cda11568-kube-api-access-rtljx\" (UniqueName: \"kubernetes.io/projected/9040ce87-14fd-4f52-acc2-3368cda11568-kube-api-access-rtljx\") pod \"9040ce87-14fd-4f52-acc2-3368cda11568\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " Oct 31 01:12:41.393500 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.393340 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9040ce87-14fd-4f52-acc2-3368cda11568-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9040ce87-14fd-4f52-acc2-3368cda11568-kube-rbac-proxy-sar-config\") pod \"9040ce87-14fd-4f52-acc2-3368cda11568\" (UID: \"9040ce87-14fd-4f52-acc2-3368cda11568\") " Oct 31 01:12:41.393612 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.393557 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9040ce87-14fd-4f52-acc2-3368cda11568-cabundle-cert" pod "9040ce87-14fd-4f52-acc2-3368cda11568" (UID: "9040ce87-14fd-4f52-acc2-3368cda11568"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:12:41.393612 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.393562 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9040ce87-14fd-4f52-acc2-3368cda11568-kserve-provision-location" pod "9040ce87-14fd-4f52-acc2-3368cda11568" (UID: "9040ce87-14fd-4f52-acc2-3368cda11568"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:12:41.393803 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.393784 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9040ce87-14fd-4f52-acc2-3368cda11568-kube-rbac-proxy-sar-config" pod "9040ce87-14fd-4f52-acc2-3368cda11568" (UID: "9040ce87-14fd-4f52-acc2-3368cda11568"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:12:41.395386 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.395367 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9040ce87-14fd-4f52-acc2-3368cda11568-proxy-tls" pod "9040ce87-14fd-4f52-acc2-3368cda11568" (UID: "9040ce87-14fd-4f52-acc2-3368cda11568"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:12:41.395484 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.395461 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9040ce87-14fd-4f52-acc2-3368cda11568-kube-api-access-rtljx" pod "9040ce87-14fd-4f52-acc2-3368cda11568" (UID: "9040ce87-14fd-4f52-acc2-3368cda11568"). InnerVolumeSpecName "kube-api-access-rtljx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:12:41.494287 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.494257 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9040ce87-14fd-4f52-acc2-3368cda11568-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:12:41.494287 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.494283 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtljx\" (UniqueName: \"kubernetes.io/projected/9040ce87-14fd-4f52-acc2-3368cda11568-kube-api-access-rtljx\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:12:41.494287 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.494293 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9040ce87-14fd-4f52-acc2-3368cda11568-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:12:41.494519 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.494307 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9040ce87-14fd-4f52-acc2-3368cda11568-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:12:41.494519 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.494320 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9040ce87-14fd-4f52-acc2-3368cda11568-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:12:41.856907 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.856873 5939 generic.go:356] "Generic (PLEG): container finished" podID="9040ce87-14fd-4f52-acc2-3368cda11568" containerID="bcd79fe72993e6b7fc3e89d1e37677e3bc215d1b6704bf84c05bd32ed07a3183" exitCode=0 Oct 31 01:12:41.857095 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.856930 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" Oct 31 01:12:41.857095 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.856942 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" event={"ID":"9040ce87-14fd-4f52-acc2-3368cda11568","Type":"ContainerDied","Data":"bcd79fe72993e6b7fc3e89d1e37677e3bc215d1b6704bf84c05bd32ed07a3183"} Oct 31 01:12:41.857095 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.856991 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh" event={"ID":"9040ce87-14fd-4f52-acc2-3368cda11568","Type":"ContainerDied","Data":"b8f48596fa46017cdfa2f7a84e4350f45801349f37d6e0d771c3cd9f0d4db2da"} Oct 31 01:12:41.857095 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.857019 5939 scope.go:122] "RemoveContainer" containerID="8fc2f49dcf94e9963162d32b5e405c258983c3c44440cc02b93b7c5456fdf579" Oct 31 01:12:41.865430 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.865412 5939 scope.go:122] "RemoveContainer" containerID="bcd79fe72993e6b7fc3e89d1e37677e3bc215d1b6704bf84c05bd32ed07a3183" Oct 31 01:12:41.873155 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.873136 5939 scope.go:122] "RemoveContainer" containerID="2e6d81bd4e789b679f9630d46bba182b87b9dafe485417bcc4af51813340bcd0" Oct 31 01:12:41.880946 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.880921 5939 scope.go:122] "RemoveContainer" containerID="8fc2f49dcf94e9963162d32b5e405c258983c3c44440cc02b93b7c5456fdf579" Oct 31 01:12:41.881308 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:12:41.881282 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc2f49dcf94e9963162d32b5e405c258983c3c44440cc02b93b7c5456fdf579\": container with ID starting with 8fc2f49dcf94e9963162d32b5e405c258983c3c44440cc02b93b7c5456fdf579 not found: ID does not exist" containerID="8fc2f49dcf94e9963162d32b5e405c258983c3c44440cc02b93b7c5456fdf579" Oct 31 01:12:41.881428 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.881317 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc2f49dcf94e9963162d32b5e405c258983c3c44440cc02b93b7c5456fdf579"} err="failed to get container status \"8fc2f49dcf94e9963162d32b5e405c258983c3c44440cc02b93b7c5456fdf579\": rpc error: code = NotFound desc = could not find container \"8fc2f49dcf94e9963162d32b5e405c258983c3c44440cc02b93b7c5456fdf579\": container with ID starting with 8fc2f49dcf94e9963162d32b5e405c258983c3c44440cc02b93b7c5456fdf579 not found: ID does not exist" Oct 31 01:12:41.881428 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.881335 5939 scope.go:122] "RemoveContainer" containerID="bcd79fe72993e6b7fc3e89d1e37677e3bc215d1b6704bf84c05bd32ed07a3183" Oct 31 01:12:41.882080 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:12:41.882053 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd79fe72993e6b7fc3e89d1e37677e3bc215d1b6704bf84c05bd32ed07a3183\": container with ID starting with bcd79fe72993e6b7fc3e89d1e37677e3bc215d1b6704bf84c05bd32ed07a3183 not found: ID does not exist" containerID="bcd79fe72993e6b7fc3e89d1e37677e3bc215d1b6704bf84c05bd32ed07a3183" Oct 31 01:12:41.882174 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.882090 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd79fe72993e6b7fc3e89d1e37677e3bc215d1b6704bf84c05bd32ed07a3183"} err="failed to get container status \"bcd79fe72993e6b7fc3e89d1e37677e3bc215d1b6704bf84c05bd32ed07a3183\": rpc error: code = NotFound desc = could not find container \"bcd79fe72993e6b7fc3e89d1e37677e3bc215d1b6704bf84c05bd32ed07a3183\": container with ID starting with bcd79fe72993e6b7fc3e89d1e37677e3bc215d1b6704bf84c05bd32ed07a3183 not found: ID does not exist" Oct 31 01:12:41.882174 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.882117 5939 scope.go:122] "RemoveContainer" containerID="2e6d81bd4e789b679f9630d46bba182b87b9dafe485417bcc4af51813340bcd0" Oct 31 01:12:41.882420 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:12:41.882401 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e6d81bd4e789b679f9630d46bba182b87b9dafe485417bcc4af51813340bcd0\": container with ID starting with 2e6d81bd4e789b679f9630d46bba182b87b9dafe485417bcc4af51813340bcd0 not found: ID does not exist" containerID="2e6d81bd4e789b679f9630d46bba182b87b9dafe485417bcc4af51813340bcd0" Oct 31 01:12:41.882467 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.882427 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e6d81bd4e789b679f9630d46bba182b87b9dafe485417bcc4af51813340bcd0"} err="failed to get container status \"2e6d81bd4e789b679f9630d46bba182b87b9dafe485417bcc4af51813340bcd0\": rpc error: code = NotFound desc = could not find container \"2e6d81bd4e789b679f9630d46bba182b87b9dafe485417bcc4af51813340bcd0\": container with ID starting with 2e6d81bd4e789b679f9630d46bba182b87b9dafe485417bcc4af51813340bcd0 not found: ID does not exist" Oct 31 01:12:41.882908 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.882888 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh"] Oct 31 01:12:41.888736 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:41.888715 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-create-v2-predictor-5d6fb757bb-khqxh"] Oct 31 01:12:43.007330 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.007293 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq"] Oct 31 01:12:43.007840 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.007810 5939 cpu_manager.go:414] "containerMap: removing container" podUID="0b1964b9-0445-4de4-8404-495044b7c020" containerName="kserve-container" Oct 31 01:12:43.007840 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.007829 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b1964b9-0445-4de4-8404-495044b7c020" containerName="kserve-container" Oct 31 01:12:43.007963 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.007848 5939 cpu_manager.go:414] "containerMap: removing container" podUID="0b1964b9-0445-4de4-8404-495044b7c020" containerName="kube-rbac-proxy" Oct 31 01:12:43.007963 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.007855 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b1964b9-0445-4de4-8404-495044b7c020" containerName="kube-rbac-proxy" Oct 31 01:12:43.007963 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.007876 5939 cpu_manager.go:414] "containerMap: removing container" podUID="0b1964b9-0445-4de4-8404-495044b7c020" containerName="storage-initializer" Oct 31 01:12:43.007963 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.007884 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b1964b9-0445-4de4-8404-495044b7c020" containerName="storage-initializer" Oct 31 01:12:43.007963 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.007902 5939 cpu_manager.go:414] "containerMap: removing container" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kserve-container" Oct 31 01:12:43.007963 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.007910 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kserve-container" Oct 31 01:12:43.007963 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.007925 5939 cpu_manager.go:414] "containerMap: removing container" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="storage-initializer" Oct 31 01:12:43.007963 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.007933 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="storage-initializer" Oct 31 01:12:43.007963 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.007943 5939 cpu_manager.go:414] "containerMap: removing container" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="storage-initializer" Oct 31 01:12:43.007963 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.007950 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="storage-initializer" Oct 31 01:12:43.008456 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.007967 5939 cpu_manager.go:414] "containerMap: removing container" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kube-rbac-proxy" Oct 31 01:12:43.008456 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.007974 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kube-rbac-proxy" Oct 31 01:12:43.008456 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.007992 5939 cpu_manager.go:414] "containerMap: removing container" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kube-rbac-proxy" Oct 31 01:12:43.008456 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.008000 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kube-rbac-proxy" Oct 31 01:12:43.008456 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.008011 5939 cpu_manager.go:414] "containerMap: removing container" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kserve-container" Oct 31 01:12:43.008456 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.008018 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kserve-container" Oct 31 01:12:43.008456 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.008094 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kube-rbac-proxy" Oct 31 01:12:43.008456 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.008105 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kserve-container" Oct 31 01:12:43.008456 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.008116 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b1964b9-0445-4de4-8404-495044b7c020" containerName="kserve-container" Oct 31 01:12:43.008456 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.008127 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b1964b9-0445-4de4-8404-495044b7c020" containerName="kube-rbac-proxy" Oct 31 01:12:43.008456 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.008138 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" containerName="kserve-container" Oct 31 01:12:43.008456 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.008149 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf9f156-749c-4050-a0e7-1efabf7550a2" containerName="kube-rbac-proxy" Oct 31 01:12:43.013138 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.013114 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:43.024610 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.024588 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq"] Oct 31 01:12:43.055587 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.055565 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\"" Oct 31 01:12:43.055814 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.055791 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 01:12:43.055923 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.055796 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\"" Oct 31 01:12:43.055923 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.055817 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\"" Oct 31 01:12:43.055923 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.055820 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\"" Oct 31 01:12:43.055923 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.055828 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\"" Oct 31 01:12:43.055923 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.055892 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-promote-predictor-serving-cert\"" Oct 31 01:12:43.104842 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.104811 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24d23233-e4d6-4247-8dde-5557c4b042da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-59b8558b77-5tkzq\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:43.104842 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.104845 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24d23233-e4d6-4247-8dde-5557c4b042da-kserve-provision-location\") pod \"isvc-canary-promote-predictor-59b8558b77-5tkzq\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:43.105057 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.104874 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24d23233-e4d6-4247-8dde-5557c4b042da-cabundle-cert\") pod \"isvc-canary-promote-predictor-59b8558b77-5tkzq\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:43.105057 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.104920 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbvfv\" (UniqueName: \"kubernetes.io/projected/24d23233-e4d6-4247-8dde-5557c4b042da-kube-api-access-pbvfv\") pod \"isvc-canary-promote-predictor-59b8558b77-5tkzq\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:43.105057 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.104976 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24d23233-e4d6-4247-8dde-5557c4b042da-proxy-tls\") pod \"isvc-canary-promote-predictor-59b8558b77-5tkzq\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:43.206376 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.206337 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbvfv\" (UniqueName: \"kubernetes.io/projected/24d23233-e4d6-4247-8dde-5557c4b042da-kube-api-access-pbvfv\") pod \"isvc-canary-promote-predictor-59b8558b77-5tkzq\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:43.206527 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.206392 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24d23233-e4d6-4247-8dde-5557c4b042da-proxy-tls\") pod \"isvc-canary-promote-predictor-59b8558b77-5tkzq\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:43.206527 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.206491 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24d23233-e4d6-4247-8dde-5557c4b042da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-59b8558b77-5tkzq\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:43.206527 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.206514 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24d23233-e4d6-4247-8dde-5557c4b042da-kserve-provision-location\") pod \"isvc-canary-promote-predictor-59b8558b77-5tkzq\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:43.206650 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.206557 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24d23233-e4d6-4247-8dde-5557c4b042da-cabundle-cert\") pod \"isvc-canary-promote-predictor-59b8558b77-5tkzq\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:43.207033 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.207010 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24d23233-e4d6-4247-8dde-5557c4b042da-kserve-provision-location\") pod \"isvc-canary-promote-predictor-59b8558b77-5tkzq\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:43.207227 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.207196 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24d23233-e4d6-4247-8dde-5557c4b042da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-59b8558b77-5tkzq\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:43.207296 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.207264 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24d23233-e4d6-4247-8dde-5557c4b042da-cabundle-cert\") pod \"isvc-canary-promote-predictor-59b8558b77-5tkzq\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:43.208935 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.208915 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24d23233-e4d6-4247-8dde-5557c4b042da-proxy-tls\") pod \"isvc-canary-promote-predictor-59b8558b77-5tkzq\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:43.221608 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.221584 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbvfv\" (UniqueName: \"kubernetes.io/projected/24d23233-e4d6-4247-8dde-5557c4b042da-kube-api-access-pbvfv\") pod \"isvc-canary-promote-predictor-59b8558b77-5tkzq\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:43.330864 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.330781 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:43.612330 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.612285 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq"] Oct 31 01:12:43.615510 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:12:43.615486 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d23233_e4d6_4247_8dde_5557c4b042da.slice/crio-2bc301a6ecad1c05ea3f9fd6c460146912f302145b1abc3663f72361439afc14 WatchSource:0}: Error finding container 2bc301a6ecad1c05ea3f9fd6c460146912f302145b1abc3663f72361439afc14: Status 404 returned error can't find the container with id 2bc301a6ecad1c05ea3f9fd6c460146912f302145b1abc3663f72361439afc14 Oct 31 01:12:43.652469 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.652442 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9040ce87-14fd-4f52-acc2-3368cda11568" path="/var/lib/kubelet/pods/9040ce87-14fd-4f52-acc2-3368cda11568/volumes" Oct 31 01:12:43.869004 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.868912 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" event={"ID":"24d23233-e4d6-4247-8dde-5557c4b042da","Type":"ContainerStarted","Data":"eded4b58759319dd54e52e9882e89e565fe3c5334e37c0f936693456180d031b"} Oct 31 01:12:43.869004 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:43.868977 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" event={"ID":"24d23233-e4d6-4247-8dde-5557c4b042da","Type":"ContainerStarted","Data":"2bc301a6ecad1c05ea3f9fd6c460146912f302145b1abc3663f72361439afc14"} Oct 31 01:12:47.888748 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:47.888717 5939 generic.go:356] "Generic (PLEG): container finished" podID="24d23233-e4d6-4247-8dde-5557c4b042da" containerID="eded4b58759319dd54e52e9882e89e565fe3c5334e37c0f936693456180d031b" exitCode=0 Oct 31 01:12:47.889198 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:47.888780 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" event={"ID":"24d23233-e4d6-4247-8dde-5557c4b042da","Type":"ContainerDied","Data":"eded4b58759319dd54e52e9882e89e565fe3c5334e37c0f936693456180d031b"} Oct 31 01:12:48.894718 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:48.894681 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" event={"ID":"24d23233-e4d6-4247-8dde-5557c4b042da","Type":"ContainerStarted","Data":"8b7f7929e6f16180408f9eeab6e4767993285a2a57fdb987b796509d7de54cc7"} Oct 31 01:12:48.895116 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:48.894748 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" event={"ID":"24d23233-e4d6-4247-8dde-5557c4b042da","Type":"ContainerStarted","Data":"c0f7af09f5979d54cf6ffd4823d8339b21f6555d3dead6b5280b7e0f8a828e1b"} Oct 31 01:12:48.895116 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:48.895027 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:48.895205 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:48.895179 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:48.896232 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:48.896208 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused" Oct 31 01:12:48.924181 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:48.924141 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" podStartSLOduration=2.763207873 podStartE2EDuration="6.924130538s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.160922665s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:48.922192333 +0000 UTC m=+2581.411467512" watchObservedRunningTime="2026-10-31 01:12:48.924130538 +0000 UTC m=+2581.413405716" Oct 31 01:12:49.899058 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:49.899020 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused" Oct 31 01:12:54.903630 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:54.903595 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:12:54.904192 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:12:54.904163 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused" Oct 31 01:13:00.508920 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:13:00.508885 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:13:04.904566 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:04.904518 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused" Oct 31 01:13:14.905082 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:14.905042 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused" Oct 31 01:13:24.904475 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:24.904431 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused" Oct 31 01:13:26.921225 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:26.921191 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs"] Oct 31 01:13:26.925210 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:26.925187 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:26.931360 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:26.931337 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\"" Oct 31 01:13:26.931360 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:26.931351 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 01:13:26.932168 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:26.932124 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"seaweedfs-s3-creds\"" Oct 31 01:13:26.932168 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:26.932138 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\"" Oct 31 01:13:26.932168 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:26.932148 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 01:13:26.932168 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:26.932166 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 01:13:26.932435 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:26.932171 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 01:13:26.939085 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:26.939065 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs"] Oct 31 01:13:26.986113 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:26.986078 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msz2l\" (UniqueName: \"kubernetes.io/projected/44cdfce8-192f-45a3-96d8-637027ad8a08-kube-api-access-msz2l\") pod \"isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:26.986289 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:26.986125 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44cdfce8-192f-45a3-96d8-637027ad8a08-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:26.986360 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:26.986318 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44cdfce8-192f-45a3-96d8-637027ad8a08-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:26.986360 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:26.986348 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44cdfce8-192f-45a3-96d8-637027ad8a08-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:26.986473 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:26.986380 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44cdfce8-192f-45a3-96d8-637027ad8a08-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:27.086834 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:27.086798 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44cdfce8-192f-45a3-96d8-637027ad8a08-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:27.086834 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:27.086839 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44cdfce8-192f-45a3-96d8-637027ad8a08-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:27.087079 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:27.086867 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44cdfce8-192f-45a3-96d8-637027ad8a08-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:27.087079 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:27.086926 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msz2l\" (UniqueName: \"kubernetes.io/projected/44cdfce8-192f-45a3-96d8-637027ad8a08-kube-api-access-msz2l\") pod \"isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:27.087079 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:27.086951 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44cdfce8-192f-45a3-96d8-637027ad8a08-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:27.087386 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:27.087364 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44cdfce8-192f-45a3-96d8-637027ad8a08-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:27.087565 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:27.087544 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44cdfce8-192f-45a3-96d8-637027ad8a08-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:27.087609 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:27.087549 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44cdfce8-192f-45a3-96d8-637027ad8a08-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:27.089327 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:27.089307 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44cdfce8-192f-45a3-96d8-637027ad8a08-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:27.101816 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:27.101792 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msz2l\" (UniqueName: \"kubernetes.io/projected/44cdfce8-192f-45a3-96d8-637027ad8a08-kube-api-access-msz2l\") pod \"isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:27.243205 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:27.243121 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:27.527380 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:27.527352 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs"] Oct 31 01:13:27.529938 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:13:27.529906 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44cdfce8_192f_45a3_96d8_637027ad8a08.slice/crio-f6cddccb4dfebea591b33ebd2fa93bd525012b6c134a19c72feaf53cfc4e28cf WatchSource:0}: Error finding container f6cddccb4dfebea591b33ebd2fa93bd525012b6c134a19c72feaf53cfc4e28cf: Status 404 returned error can't find the container with id f6cddccb4dfebea591b33ebd2fa93bd525012b6c134a19c72feaf53cfc4e28cf Oct 31 01:13:28.055992 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:28.055960 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" event={"ID":"44cdfce8-192f-45a3-96d8-637027ad8a08","Type":"ContainerStarted","Data":"8c0d7432f6e8eee2bfde354ca2b4b6748d258f95592c4b97c1a8e2839c149a0c"} Oct 31 01:13:28.056387 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:28.056014 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" event={"ID":"44cdfce8-192f-45a3-96d8-637027ad8a08","Type":"ContainerStarted","Data":"f6cddccb4dfebea591b33ebd2fa93bd525012b6c134a19c72feaf53cfc4e28cf"} Oct 31 01:13:30.508683 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:13:30.508630 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:13:32.074574 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:32.074539 5939 generic.go:356] "Generic (PLEG): container finished" podID="44cdfce8-192f-45a3-96d8-637027ad8a08" containerID="8c0d7432f6e8eee2bfde354ca2b4b6748d258f95592c4b97c1a8e2839c149a0c" exitCode=0 Oct 31 01:13:32.074982 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:32.074614 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" event={"ID":"44cdfce8-192f-45a3-96d8-637027ad8a08","Type":"ContainerDied","Data":"8c0d7432f6e8eee2bfde354ca2b4b6748d258f95592c4b97c1a8e2839c149a0c"} Oct 31 01:13:33.080941 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:33.080906 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" event={"ID":"44cdfce8-192f-45a3-96d8-637027ad8a08","Type":"ContainerStarted","Data":"a87101051ab9f751ec6b735a04c99d3eae02e4ea6a38e69e7f4be96c53247698"} Oct 31 01:13:33.081436 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:33.080977 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" event={"ID":"44cdfce8-192f-45a3-96d8-637027ad8a08","Type":"ContainerStarted","Data":"b821c31d8684f2a568b8a28ea39410e3f0154c90f6b7d0ad1f4ab67ff9bb39f4"} Oct 31 01:13:33.081436 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:33.081193 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:33.111412 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:33.111368 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" podStartSLOduration=3.389679106 podStartE2EDuration="7.111356291s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.721677185s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:33.109452821 +0000 UTC m=+2625.598728002" watchObservedRunningTime="2026-10-31 01:13:33.111356291 +0000 UTC m=+2625.600631464" Oct 31 01:13:34.085719 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:34.085690 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:34.904527 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:34.904488 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused" Oct 31 01:13:40.101761 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:40.101730 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:13:44.904406 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:44.904367 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused" Oct 31 01:13:54.904485 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:13:54.904446 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused" Oct 31 01:14:00.509497 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:14:00.509462 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:14:04.904191 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:04.904145 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused" Oct 31 01:14:05.406806 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:05.406771 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq"] Oct 31 01:14:05.407132 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:05.407085 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kserve-container" containerID="cri-o://c0f7af09f5979d54cf6ffd4823d8339b21f6555d3dead6b5280b7e0f8a828e1b" gracePeriod=30 Oct 31 01:14:05.407271 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:05.407141 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kube-rbac-proxy" containerID="cri-o://8b7f7929e6f16180408f9eeab6e4767993285a2a57fdb987b796509d7de54cc7" gracePeriod=30 Oct 31 01:14:05.407334 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:05.407308 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused" Oct 31 01:14:06.230195 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:06.230086 5939 generic.go:356] "Generic (PLEG): container finished" podID="24d23233-e4d6-4247-8dde-5557c4b042da" containerID="8b7f7929e6f16180408f9eeab6e4767993285a2a57fdb987b796509d7de54cc7" exitCode=0 Oct 31 01:14:06.230195 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:06.230168 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" event={"ID":"24d23233-e4d6-4247-8dde-5557c4b042da","Type":"ContainerDied","Data":"8b7f7929e6f16180408f9eeab6e4767993285a2a57fdb987b796509d7de54cc7"} Oct 31 01:14:09.900153 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:09.900107 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused" Oct 31 01:14:10.114148 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:10.114097 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" podUID="44cdfce8-192f-45a3-96d8-637027ad8a08" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400" Oct 31 01:14:10.650505 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:10.650478 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:14:10.758467 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:10.758437 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d23233-e4d6-4247-8dde-5557c4b042da-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24d23233-e4d6-4247-8dde-5557c4b042da-kserve-provision-location\") pod \"24d23233-e4d6-4247-8dde-5557c4b042da\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " Oct 31 01:14:10.758660 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:10.758478 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24d23233-e4d6-4247-8dde-5557c4b042da-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24d23233-e4d6-4247-8dde-5557c4b042da-kube-rbac-proxy-sar-config\") pod \"24d23233-e4d6-4247-8dde-5557c4b042da\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " Oct 31 01:14:10.758660 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:10.758522 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d23233-e4d6-4247-8dde-5557c4b042da-kube-api-access-pbvfv\" (UniqueName: \"kubernetes.io/projected/24d23233-e4d6-4247-8dde-5557c4b042da-kube-api-access-pbvfv\") pod \"24d23233-e4d6-4247-8dde-5557c4b042da\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " Oct 31 01:14:10.758660 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:10.758628 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24d23233-e4d6-4247-8dde-5557c4b042da-proxy-tls\" (UniqueName: \"kubernetes.io/secret/24d23233-e4d6-4247-8dde-5557c4b042da-proxy-tls\") pod \"24d23233-e4d6-4247-8dde-5557c4b042da\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " Oct 31 01:14:10.758870 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:10.758696 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24d23233-e4d6-4247-8dde-5557c4b042da-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24d23233-e4d6-4247-8dde-5557c4b042da-cabundle-cert\") pod \"24d23233-e4d6-4247-8dde-5557c4b042da\" (UID: \"24d23233-e4d6-4247-8dde-5557c4b042da\") " Oct 31 01:14:10.758929 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:10.758861 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d23233-e4d6-4247-8dde-5557c4b042da-kserve-provision-location" pod "24d23233-e4d6-4247-8dde-5557c4b042da" (UID: "24d23233-e4d6-4247-8dde-5557c4b042da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:10.758929 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:10.758888 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d23233-e4d6-4247-8dde-5557c4b042da-kube-rbac-proxy-sar-config" pod "24d23233-e4d6-4247-8dde-5557c4b042da" (UID: "24d23233-e4d6-4247-8dde-5557c4b042da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:10.759269 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:10.759240 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d23233-e4d6-4247-8dde-5557c4b042da-cabundle-cert" pod "24d23233-e4d6-4247-8dde-5557c4b042da" (UID: "24d23233-e4d6-4247-8dde-5557c4b042da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:10.759716 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:10.759695 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24d23233-e4d6-4247-8dde-5557c4b042da-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:10.759806 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:10.759723 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24d23233-e4d6-4247-8dde-5557c4b042da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:10.759806 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:10.759741 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24d23233-e4d6-4247-8dde-5557c4b042da-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:10.760815 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:10.760794 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d23233-e4d6-4247-8dde-5557c4b042da-proxy-tls" pod "24d23233-e4d6-4247-8dde-5557c4b042da" (UID: "24d23233-e4d6-4247-8dde-5557c4b042da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:10.760933 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:10.760917 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d23233-e4d6-4247-8dde-5557c4b042da-kube-api-access-pbvfv" pod "24d23233-e4d6-4247-8dde-5557c4b042da" (UID: "24d23233-e4d6-4247-8dde-5557c4b042da"). InnerVolumeSpecName "kube-api-access-pbvfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:10.860710 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:10.860682 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbvfv\" (UniqueName: \"kubernetes.io/projected/24d23233-e4d6-4247-8dde-5557c4b042da-kube-api-access-pbvfv\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:10.860876 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:10.860713 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24d23233-e4d6-4247-8dde-5557c4b042da-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:11.259227 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:11.259115 5939 generic.go:356] "Generic (PLEG): container finished" podID="24d23233-e4d6-4247-8dde-5557c4b042da" containerID="c0f7af09f5979d54cf6ffd4823d8339b21f6555d3dead6b5280b7e0f8a828e1b" exitCode=139 Oct 31 01:14:11.259227 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:11.259179 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" Oct 31 01:14:11.259227 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:11.259208 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" event={"ID":"24d23233-e4d6-4247-8dde-5557c4b042da","Type":"ContainerDied","Data":"c0f7af09f5979d54cf6ffd4823d8339b21f6555d3dead6b5280b7e0f8a828e1b"} Oct 31 01:14:11.259931 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:11.259317 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq" event={"ID":"24d23233-e4d6-4247-8dde-5557c4b042da","Type":"ContainerDied","Data":"2bc301a6ecad1c05ea3f9fd6c460146912f302145b1abc3663f72361439afc14"} Oct 31 01:14:11.259931 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:11.259359 5939 scope.go:122] "RemoveContainer" containerID="8b7f7929e6f16180408f9eeab6e4767993285a2a57fdb987b796509d7de54cc7" Oct 31 01:14:11.268851 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:11.268829 5939 scope.go:122] "RemoveContainer" containerID="c0f7af09f5979d54cf6ffd4823d8339b21f6555d3dead6b5280b7e0f8a828e1b" Oct 31 01:14:11.276630 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:11.276611 5939 scope.go:122] "RemoveContainer" containerID="eded4b58759319dd54e52e9882e89e565fe3c5334e37c0f936693456180d031b" Oct 31 01:14:11.284240 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:11.284222 5939 scope.go:122] "RemoveContainer" containerID="8b7f7929e6f16180408f9eeab6e4767993285a2a57fdb987b796509d7de54cc7" Oct 31 01:14:11.284520 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:14:11.284494 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b7f7929e6f16180408f9eeab6e4767993285a2a57fdb987b796509d7de54cc7\": container with ID starting with 8b7f7929e6f16180408f9eeab6e4767993285a2a57fdb987b796509d7de54cc7 not found: ID does not exist" containerID="8b7f7929e6f16180408f9eeab6e4767993285a2a57fdb987b796509d7de54cc7" Oct 31 01:14:11.284573 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:11.284529 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b7f7929e6f16180408f9eeab6e4767993285a2a57fdb987b796509d7de54cc7"} err="failed to get container status \"8b7f7929e6f16180408f9eeab6e4767993285a2a57fdb987b796509d7de54cc7\": rpc error: code = NotFound desc = could not find container \"8b7f7929e6f16180408f9eeab6e4767993285a2a57fdb987b796509d7de54cc7\": container with ID starting with 8b7f7929e6f16180408f9eeab6e4767993285a2a57fdb987b796509d7de54cc7 not found: ID does not exist" Oct 31 01:14:11.284573 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:11.284548 5939 scope.go:122] "RemoveContainer" containerID="c0f7af09f5979d54cf6ffd4823d8339b21f6555d3dead6b5280b7e0f8a828e1b" Oct 31 01:14:11.284860 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:14:11.284825 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f7af09f5979d54cf6ffd4823d8339b21f6555d3dead6b5280b7e0f8a828e1b\": container with ID starting with c0f7af09f5979d54cf6ffd4823d8339b21f6555d3dead6b5280b7e0f8a828e1b not found: ID does not exist" containerID="c0f7af09f5979d54cf6ffd4823d8339b21f6555d3dead6b5280b7e0f8a828e1b" Oct 31 01:14:11.284860 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:11.284847 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f7af09f5979d54cf6ffd4823d8339b21f6555d3dead6b5280b7e0f8a828e1b"} err="failed to get container status \"c0f7af09f5979d54cf6ffd4823d8339b21f6555d3dead6b5280b7e0f8a828e1b\": rpc error: code = NotFound desc = could not find container \"c0f7af09f5979d54cf6ffd4823d8339b21f6555d3dead6b5280b7e0f8a828e1b\": container with ID starting with c0f7af09f5979d54cf6ffd4823d8339b21f6555d3dead6b5280b7e0f8a828e1b not found: ID does not exist" Oct 31 01:14:11.284860 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:11.284861 5939 scope.go:122] "RemoveContainer" containerID="eded4b58759319dd54e52e9882e89e565fe3c5334e37c0f936693456180d031b" Oct 31 01:14:11.285619 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:14:11.285141 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eded4b58759319dd54e52e9882e89e565fe3c5334e37c0f936693456180d031b\": container with ID starting with eded4b58759319dd54e52e9882e89e565fe3c5334e37c0f936693456180d031b not found: ID does not exist" containerID="eded4b58759319dd54e52e9882e89e565fe3c5334e37c0f936693456180d031b" Oct 31 01:14:11.285619 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:11.285170 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eded4b58759319dd54e52e9882e89e565fe3c5334e37c0f936693456180d031b"} err="failed to get container status \"eded4b58759319dd54e52e9882e89e565fe3c5334e37c0f936693456180d031b\": rpc error: code = NotFound desc = could not find container \"eded4b58759319dd54e52e9882e89e565fe3c5334e37c0f936693456180d031b\": container with ID starting with eded4b58759319dd54e52e9882e89e565fe3c5334e37c0f936693456180d031b not found: ID does not exist" Oct 31 01:14:11.286519 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:11.286497 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq"] Oct 31 01:14:11.293093 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:11.293074 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-predictor-59b8558b77-5tkzq"] Oct 31 01:14:11.652393 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:11.652359 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" path="/var/lib/kubelet/pods/24d23233-e4d6-4247-8dde-5557c4b042da/volumes" Oct 31 01:14:13.278138 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.278104 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl"] Oct 31 01:14:13.278624 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.278606 5939 cpu_manager.go:414] "containerMap: removing container" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kserve-container" Oct 31 01:14:13.278729 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.278628 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kserve-container" Oct 31 01:14:13.278729 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.278685 5939 cpu_manager.go:414] "containerMap: removing container" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="storage-initializer" Oct 31 01:14:13.278729 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.278694 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="storage-initializer" Oct 31 01:14:13.278729 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.278711 5939 cpu_manager.go:414] "containerMap: removing container" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kube-rbac-proxy" Oct 31 01:14:13.278729 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.278718 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kube-rbac-proxy" Oct 31 01:14:13.278982 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.278809 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kube-rbac-proxy" Oct 31 01:14:13.278982 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.278827 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d23233-e4d6-4247-8dde-5557c4b042da" containerName="kserve-container" Oct 31 01:14:13.283618 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.283595 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:13.293807 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.293780 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\"" Oct 31 01:14:13.293950 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.293824 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\"" Oct 31 01:14:13.293950 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.293836 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\"" Oct 31 01:14:13.293950 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.293826 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\"" Oct 31 01:14:13.293950 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.293931 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-promote-v2-predictor-serving-cert\"" Oct 31 01:14:13.294162 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.293974 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 01:14:13.294714 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.294681 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\"" Oct 31 01:14:13.295105 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.295089 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl"] Oct 31 01:14:13.383680 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.383642 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad243c40-ad56-4462-b9c2-889d82e7c151-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-89rkl\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:13.383879 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.383763 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad243c40-ad56-4462-b9c2-889d82e7c151-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-89rkl\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:13.383928 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.383872 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad243c40-ad56-4462-b9c2-889d82e7c151-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-89rkl\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:13.383971 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.383926 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8q68\" (UniqueName: \"kubernetes.io/projected/ad243c40-ad56-4462-b9c2-889d82e7c151-kube-api-access-g8q68\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-89rkl\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:13.384007 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.383971 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad243c40-ad56-4462-b9c2-889d82e7c151-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-89rkl\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:13.484725 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.484685 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad243c40-ad56-4462-b9c2-889d82e7c151-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-89rkl\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:13.484929 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.484748 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad243c40-ad56-4462-b9c2-889d82e7c151-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-89rkl\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:13.484929 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.484794 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8q68\" (UniqueName: \"kubernetes.io/projected/ad243c40-ad56-4462-b9c2-889d82e7c151-kube-api-access-g8q68\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-89rkl\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:13.484929 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.484813 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad243c40-ad56-4462-b9c2-889d82e7c151-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-89rkl\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:13.484929 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.484847 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad243c40-ad56-4462-b9c2-889d82e7c151-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-89rkl\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:13.485277 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.485257 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad243c40-ad56-4462-b9c2-889d82e7c151-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-89rkl\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:13.485586 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.485565 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad243c40-ad56-4462-b9c2-889d82e7c151-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-89rkl\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:13.485712 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.485688 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad243c40-ad56-4462-b9c2-889d82e7c151-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-89rkl\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:13.487330 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.487312 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad243c40-ad56-4462-b9c2-889d82e7c151-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-89rkl\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:13.499244 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.499221 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8q68\" (UniqueName: \"kubernetes.io/projected/ad243c40-ad56-4462-b9c2-889d82e7c151-kube-api-access-g8q68\") pod \"isvc-canary-promote-v2-predictor-54ff8866fd-89rkl\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:13.603253 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.603158 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:13.885587 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:13.885553 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl"] Oct 31 01:14:13.888914 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:14:13.888880 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad243c40_ad56_4462_b9c2_889d82e7c151.slice/crio-900b7a15904f9fff27c8a0b274e8a3069d6b2fd77f73a6ef564aad5ceaf5299b WatchSource:0}: Error finding container 900b7a15904f9fff27c8a0b274e8a3069d6b2fd77f73a6ef564aad5ceaf5299b: Status 404 returned error can't find the container with id 900b7a15904f9fff27c8a0b274e8a3069d6b2fd77f73a6ef564aad5ceaf5299b Oct 31 01:14:14.273329 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:14.273299 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" event={"ID":"ad243c40-ad56-4462-b9c2-889d82e7c151","Type":"ContainerStarted","Data":"c410f6e0d5db442f23b005ac68ee576b50dd3dac8a2d00870c7d98a794001000"} Oct 31 01:14:14.273488 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:14.273364 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" event={"ID":"ad243c40-ad56-4462-b9c2-889d82e7c151","Type":"ContainerStarted","Data":"900b7a15904f9fff27c8a0b274e8a3069d6b2fd77f73a6ef564aad5ceaf5299b"} Oct 31 01:14:18.292822 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:18.292784 5939 generic.go:356] "Generic (PLEG): container finished" podID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerID="c410f6e0d5db442f23b005ac68ee576b50dd3dac8a2d00870c7d98a794001000" exitCode=0 Oct 31 01:14:18.293222 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:18.292847 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" event={"ID":"ad243c40-ad56-4462-b9c2-889d82e7c151","Type":"ContainerDied","Data":"c410f6e0d5db442f23b005ac68ee576b50dd3dac8a2d00870c7d98a794001000"} Oct 31 01:14:19.299365 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:19.299329 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" event={"ID":"ad243c40-ad56-4462-b9c2-889d82e7c151","Type":"ContainerStarted","Data":"c252aaf201e8a7de8a0cf29dc8798317f86053feb3a5101321cba6114c40e3bc"} Oct 31 01:14:19.299790 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:19.299392 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" event={"ID":"ad243c40-ad56-4462-b9c2-889d82e7c151","Type":"ContainerStarted","Data":"029cc6d62e59f51f45b5dab11f4203fc62473cf7d62a515a170df932a32ade8f"} Oct 31 01:14:19.299790 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:19.299686 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:19.299905 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:19.299826 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:19.301108 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:19.301082 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused" Oct 31 01:14:19.327178 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:19.327131 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" podStartSLOduration=2.552499724 podStartE2EDuration="6.327118979s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.774619255s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:19.326482121 +0000 UTC m=+2671.815757326" watchObservedRunningTime="2026-10-31 01:14:19.327118979 +0000 UTC m=+2671.816394157" Oct 31 01:14:20.104716 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:20.104687 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:14:20.303763 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:20.303724 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused" Oct 31 01:14:25.308208 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:25.308174 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:14:25.308751 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:25.308723 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused" Oct 31 01:14:27.257113 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:27.257084 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs"] Oct 31 01:14:27.257492 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:27.257337 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" podUID="44cdfce8-192f-45a3-96d8-637027ad8a08" containerName="kserve-container" containerID="cri-o://b821c31d8684f2a568b8a28ea39410e3f0154c90f6b7d0ad1f4ab67ff9bb39f4" gracePeriod=30 Oct 31 01:14:27.257492 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:27.257388 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" podUID="44cdfce8-192f-45a3-96d8-637027ad8a08" containerName="kube-rbac-proxy" containerID="cri-o://a87101051ab9f751ec6b735a04c99d3eae02e4ea6a38e69e7f4be96c53247698" gracePeriod=30 Oct 31 01:14:28.342267 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:28.342226 5939 generic.go:356] "Generic (PLEG): container finished" podID="44cdfce8-192f-45a3-96d8-637027ad8a08" containerID="a87101051ab9f751ec6b735a04c99d3eae02e4ea6a38e69e7f4be96c53247698" exitCode=0 Oct 31 01:14:28.342267 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:28.342266 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" event={"ID":"44cdfce8-192f-45a3-96d8-637027ad8a08","Type":"ContainerDied","Data":"a87101051ab9f751ec6b735a04c99d3eae02e4ea6a38e69e7f4be96c53247698"} Oct 31 01:14:30.097717 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:30.097641 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" podUID="44cdfce8-192f-45a3-96d8-637027ad8a08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused" Oct 31 01:14:30.509062 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:14:30.509027 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:14:35.006081 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.006056 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:14:35.078745 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.078650 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44cdfce8-192f-45a3-96d8-637027ad8a08-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44cdfce8-192f-45a3-96d8-637027ad8a08-cabundle-cert\") pod \"44cdfce8-192f-45a3-96d8-637027ad8a08\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " Oct 31 01:14:35.078897 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.078806 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44cdfce8-192f-45a3-96d8-637027ad8a08-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44cdfce8-192f-45a3-96d8-637027ad8a08-kube-rbac-proxy-sar-config\") pod \"44cdfce8-192f-45a3-96d8-637027ad8a08\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " Oct 31 01:14:35.078897 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.078833 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44cdfce8-192f-45a3-96d8-637027ad8a08-proxy-tls\" (UniqueName: \"kubernetes.io/secret/44cdfce8-192f-45a3-96d8-637027ad8a08-proxy-tls\") pod \"44cdfce8-192f-45a3-96d8-637027ad8a08\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " Oct 31 01:14:35.078897 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.078892 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44cdfce8-192f-45a3-96d8-637027ad8a08-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44cdfce8-192f-45a3-96d8-637027ad8a08-kserve-provision-location\") pod \"44cdfce8-192f-45a3-96d8-637027ad8a08\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " Oct 31 01:14:35.079001 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.078939 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44cdfce8-192f-45a3-96d8-637027ad8a08-kube-api-access-msz2l\" (UniqueName: \"kubernetes.io/projected/44cdfce8-192f-45a3-96d8-637027ad8a08-kube-api-access-msz2l\") pod \"44cdfce8-192f-45a3-96d8-637027ad8a08\" (UID: \"44cdfce8-192f-45a3-96d8-637027ad8a08\") " Oct 31 01:14:35.079001 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.078961 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44cdfce8-192f-45a3-96d8-637027ad8a08-cabundle-cert" pod "44cdfce8-192f-45a3-96d8-637027ad8a08" (UID: "44cdfce8-192f-45a3-96d8-637027ad8a08"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:35.079237 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.079213 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44cdfce8-192f-45a3-96d8-637027ad8a08-kube-rbac-proxy-sar-config" pod "44cdfce8-192f-45a3-96d8-637027ad8a08" (UID: "44cdfce8-192f-45a3-96d8-637027ad8a08"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:35.079288 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.079236 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44cdfce8-192f-45a3-96d8-637027ad8a08-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.079288 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.079257 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44cdfce8-192f-45a3-96d8-637027ad8a08-kserve-provision-location" pod "44cdfce8-192f-45a3-96d8-637027ad8a08" (UID: "44cdfce8-192f-45a3-96d8-637027ad8a08"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.081125 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.081097 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44cdfce8-192f-45a3-96d8-637027ad8a08-kube-api-access-msz2l" pod "44cdfce8-192f-45a3-96d8-637027ad8a08" (UID: "44cdfce8-192f-45a3-96d8-637027ad8a08"). InnerVolumeSpecName "kube-api-access-msz2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:35.081125 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.081117 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44cdfce8-192f-45a3-96d8-637027ad8a08-proxy-tls" pod "44cdfce8-192f-45a3-96d8-637027ad8a08" (UID: "44cdfce8-192f-45a3-96d8-637027ad8a08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:35.180315 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.180270 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44cdfce8-192f-45a3-96d8-637027ad8a08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.180315 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.180308 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44cdfce8-192f-45a3-96d8-637027ad8a08-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.180315 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.180321 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44cdfce8-192f-45a3-96d8-637027ad8a08-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.180561 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.180330 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msz2l\" (UniqueName: \"kubernetes.io/projected/44cdfce8-192f-45a3-96d8-637027ad8a08-kube-api-access-msz2l\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.308868 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.308829 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused" Oct 31 01:14:35.373479 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.373444 5939 generic.go:356] "Generic (PLEG): container finished" podID="44cdfce8-192f-45a3-96d8-637027ad8a08" containerID="b821c31d8684f2a568b8a28ea39410e3f0154c90f6b7d0ad1f4ab67ff9bb39f4" exitCode=0 Oct 31 01:14:35.373652 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.373513 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" Oct 31 01:14:35.373652 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.373530 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" event={"ID":"44cdfce8-192f-45a3-96d8-637027ad8a08","Type":"ContainerDied","Data":"b821c31d8684f2a568b8a28ea39410e3f0154c90f6b7d0ad1f4ab67ff9bb39f4"} Oct 31 01:14:35.373652 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.373600 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs" event={"ID":"44cdfce8-192f-45a3-96d8-637027ad8a08","Type":"ContainerDied","Data":"f6cddccb4dfebea591b33ebd2fa93bd525012b6c134a19c72feaf53cfc4e28cf"} Oct 31 01:14:35.373652 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.373628 5939 scope.go:122] "RemoveContainer" containerID="a87101051ab9f751ec6b735a04c99d3eae02e4ea6a38e69e7f4be96c53247698" Oct 31 01:14:35.383073 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.383050 5939 scope.go:122] "RemoveContainer" containerID="b821c31d8684f2a568b8a28ea39410e3f0154c90f6b7d0ad1f4ab67ff9bb39f4" Oct 31 01:14:35.390842 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.390824 5939 scope.go:122] "RemoveContainer" containerID="8c0d7432f6e8eee2bfde354ca2b4b6748d258f95592c4b97c1a8e2839c149a0c" Oct 31 01:14:35.398969 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.398948 5939 scope.go:122] "RemoveContainer" containerID="a87101051ab9f751ec6b735a04c99d3eae02e4ea6a38e69e7f4be96c53247698" Oct 31 01:14:35.399277 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:14:35.399261 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87101051ab9f751ec6b735a04c99d3eae02e4ea6a38e69e7f4be96c53247698\": container with ID starting with a87101051ab9f751ec6b735a04c99d3eae02e4ea6a38e69e7f4be96c53247698 not found: ID does not exist" containerID="a87101051ab9f751ec6b735a04c99d3eae02e4ea6a38e69e7f4be96c53247698" Oct 31 01:14:35.399332 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.399288 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87101051ab9f751ec6b735a04c99d3eae02e4ea6a38e69e7f4be96c53247698"} err="failed to get container status \"a87101051ab9f751ec6b735a04c99d3eae02e4ea6a38e69e7f4be96c53247698\": rpc error: code = NotFound desc = could not find container \"a87101051ab9f751ec6b735a04c99d3eae02e4ea6a38e69e7f4be96c53247698\": container with ID starting with a87101051ab9f751ec6b735a04c99d3eae02e4ea6a38e69e7f4be96c53247698 not found: ID does not exist" Oct 31 01:14:35.399332 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.399308 5939 scope.go:122] "RemoveContainer" containerID="b821c31d8684f2a568b8a28ea39410e3f0154c90f6b7d0ad1f4ab67ff9bb39f4" Oct 31 01:14:35.399570 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:14:35.399552 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b821c31d8684f2a568b8a28ea39410e3f0154c90f6b7d0ad1f4ab67ff9bb39f4\": container with ID starting with b821c31d8684f2a568b8a28ea39410e3f0154c90f6b7d0ad1f4ab67ff9bb39f4 not found: ID does not exist" containerID="b821c31d8684f2a568b8a28ea39410e3f0154c90f6b7d0ad1f4ab67ff9bb39f4" Oct 31 01:14:35.399615 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.399571 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs"] Oct 31 01:14:35.399615 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.399574 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b821c31d8684f2a568b8a28ea39410e3f0154c90f6b7d0ad1f4ab67ff9bb39f4"} err="failed to get container status \"b821c31d8684f2a568b8a28ea39410e3f0154c90f6b7d0ad1f4ab67ff9bb39f4\": rpc error: code = NotFound desc = could not find container \"b821c31d8684f2a568b8a28ea39410e3f0154c90f6b7d0ad1f4ab67ff9bb39f4\": container with ID starting with b821c31d8684f2a568b8a28ea39410e3f0154c90f6b7d0ad1f4ab67ff9bb39f4 not found: ID does not exist" Oct 31 01:14:35.399705 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.399616 5939 scope.go:122] "RemoveContainer" containerID="8c0d7432f6e8eee2bfde354ca2b4b6748d258f95592c4b97c1a8e2839c149a0c" Oct 31 01:14:35.399993 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:14:35.399974 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0d7432f6e8eee2bfde354ca2b4b6748d258f95592c4b97c1a8e2839c149a0c\": container with ID starting with 8c0d7432f6e8eee2bfde354ca2b4b6748d258f95592c4b97c1a8e2839c149a0c not found: ID does not exist" containerID="8c0d7432f6e8eee2bfde354ca2b4b6748d258f95592c4b97c1a8e2839c149a0c" Oct 31 01:14:35.400070 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.399994 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0d7432f6e8eee2bfde354ca2b4b6748d258f95592c4b97c1a8e2839c149a0c"} err="failed to get container status \"8c0d7432f6e8eee2bfde354ca2b4b6748d258f95592c4b97c1a8e2839c149a0c\": rpc error: code = NotFound desc = could not find container \"8c0d7432f6e8eee2bfde354ca2b4b6748d258f95592c4b97c1a8e2839c149a0c\": container with ID starting with 8c0d7432f6e8eee2bfde354ca2b4b6748d258f95592c4b97c1a8e2839c149a0c not found: ID does not exist" Oct 31 01:14:35.407100 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.407077 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-xgboost-v2-runtime-predictor-5f9ddcd5d-5r2zs"] Oct 31 01:14:35.652608 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:35.652526 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44cdfce8-192f-45a3-96d8-637027ad8a08" path="/var/lib/kubelet/pods/44cdfce8-192f-45a3-96d8-637027ad8a08/volumes" Oct 31 01:14:45.309177 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:45.309129 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused" Oct 31 01:14:47.966761 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:47.966732 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 01:14:47.968389 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:47.968368 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 01:14:55.308985 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:14:55.308943 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused" Oct 31 01:15:00.509079 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:15:00.508999 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:15:05.308714 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:15:05.308658 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused" Oct 31 01:15:15.308850 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:15:15.308804 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused" Oct 31 01:15:25.308637 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:15:25.308590 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused" Oct 31 01:15:30.509034 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:15:30.508997 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:15:35.309314 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:15:35.309269 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused" Oct 31 01:15:45.309526 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:15:45.309492 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:15:53.669334 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:15:53.669305 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl"] Oct 31 01:15:53.669770 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:15:53.669522 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kserve-container" containerID="cri-o://029cc6d62e59f51f45b5dab11f4203fc62473cf7d62a515a170df932a32ade8f" gracePeriod=30 Oct 31 01:15:53.669770 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:15:53.669560 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kube-rbac-proxy" containerID="cri-o://c252aaf201e8a7de8a0cf29dc8798317f86053feb3a5101321cba6114c40e3bc" gracePeriod=30 Oct 31 01:15:54.684634 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:15:54.684601 5939 generic.go:356] "Generic (PLEG): container finished" podID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerID="c252aaf201e8a7de8a0cf29dc8798317f86053feb3a5101321cba6114c40e3bc" exitCode=0 Oct 31 01:15:54.685045 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:15:54.684686 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" event={"ID":"ad243c40-ad56-4462-b9c2-889d82e7c151","Type":"ContainerDied","Data":"c252aaf201e8a7de8a0cf29dc8798317f86053feb3a5101321cba6114c40e3bc"} Oct 31 01:15:55.304420 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:15:55.304383 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused" Oct 31 01:15:55.308651 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:15:55.308625 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused" Oct 31 01:16:00.304401 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:00.304354 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused" Oct 31 01:16:00.508264 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:16:00.508221 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:16:01.713153 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:01.713120 5939 generic.go:356] "Generic (PLEG): container finished" podID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerID="029cc6d62e59f51f45b5dab11f4203fc62473cf7d62a515a170df932a32ade8f" exitCode=0 Oct 31 01:16:01.713530 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:01.713160 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" event={"ID":"ad243c40-ad56-4462-b9c2-889d82e7c151","Type":"ContainerDied","Data":"029cc6d62e59f51f45b5dab11f4203fc62473cf7d62a515a170df932a32ade8f"} Oct 31 01:16:01.811868 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:01.811845 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:16:01.899518 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:01.899451 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad243c40-ad56-4462-b9c2-889d82e7c151-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad243c40-ad56-4462-b9c2-889d82e7c151-kserve-provision-location\") pod \"ad243c40-ad56-4462-b9c2-889d82e7c151\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " Oct 31 01:16:01.899652 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:01.899584 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad243c40-ad56-4462-b9c2-889d82e7c151-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad243c40-ad56-4462-b9c2-889d82e7c151-kube-rbac-proxy-sar-config\") pod \"ad243c40-ad56-4462-b9c2-889d82e7c151\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " Oct 31 01:16:01.899652 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:01.899623 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad243c40-ad56-4462-b9c2-889d82e7c151-kube-api-access-g8q68\" (UniqueName: \"kubernetes.io/projected/ad243c40-ad56-4462-b9c2-889d82e7c151-kube-api-access-g8q68\") pod \"ad243c40-ad56-4462-b9c2-889d82e7c151\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " Oct 31 01:16:01.899775 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:01.899649 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad243c40-ad56-4462-b9c2-889d82e7c151-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad243c40-ad56-4462-b9c2-889d82e7c151-cabundle-cert\") pod \"ad243c40-ad56-4462-b9c2-889d82e7c151\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " Oct 31 01:16:01.899775 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:01.899719 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad243c40-ad56-4462-b9c2-889d82e7c151-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad243c40-ad56-4462-b9c2-889d82e7c151-proxy-tls\") pod \"ad243c40-ad56-4462-b9c2-889d82e7c151\" (UID: \"ad243c40-ad56-4462-b9c2-889d82e7c151\") " Oct 31 01:16:01.899951 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:01.899857 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad243c40-ad56-4462-b9c2-889d82e7c151-kserve-provision-location" pod "ad243c40-ad56-4462-b9c2-889d82e7c151" (UID: "ad243c40-ad56-4462-b9c2-889d82e7c151"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:16:01.900014 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:01.899995 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad243c40-ad56-4462-b9c2-889d82e7c151-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:16:01.900132 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:01.900106 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad243c40-ad56-4462-b9c2-889d82e7c151-cabundle-cert" pod "ad243c40-ad56-4462-b9c2-889d82e7c151" (UID: "ad243c40-ad56-4462-b9c2-889d82e7c151"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:16:01.900182 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:01.900134 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad243c40-ad56-4462-b9c2-889d82e7c151-kube-rbac-proxy-sar-config" pod "ad243c40-ad56-4462-b9c2-889d82e7c151" (UID: "ad243c40-ad56-4462-b9c2-889d82e7c151"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:16:01.901848 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:01.901824 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad243c40-ad56-4462-b9c2-889d82e7c151-kube-api-access-g8q68" pod "ad243c40-ad56-4462-b9c2-889d82e7c151" (UID: "ad243c40-ad56-4462-b9c2-889d82e7c151"). InnerVolumeSpecName "kube-api-access-g8q68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:16:01.901962 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:01.901860 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad243c40-ad56-4462-b9c2-889d82e7c151-proxy-tls" pod "ad243c40-ad56-4462-b9c2-889d82e7c151" (UID: "ad243c40-ad56-4462-b9c2-889d82e7c151"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:16:02.001224 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:02.001195 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad243c40-ad56-4462-b9c2-889d82e7c151-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:16:02.001224 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:02.001222 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8q68\" (UniqueName: \"kubernetes.io/projected/ad243c40-ad56-4462-b9c2-889d82e7c151-kube-api-access-g8q68\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:16:02.001410 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:02.001234 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad243c40-ad56-4462-b9c2-889d82e7c151-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:16:02.001410 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:02.001244 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad243c40-ad56-4462-b9c2-889d82e7c151-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:16:02.719359 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:02.719320 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" event={"ID":"ad243c40-ad56-4462-b9c2-889d82e7c151","Type":"ContainerDied","Data":"900b7a15904f9fff27c8a0b274e8a3069d6b2fd77f73a6ef564aad5ceaf5299b"} Oct 31 01:16:02.719359 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:02.719347 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl" Oct 31 01:16:02.719859 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:02.719384 5939 scope.go:122] "RemoveContainer" containerID="c252aaf201e8a7de8a0cf29dc8798317f86053feb3a5101321cba6114c40e3bc" Oct 31 01:16:02.728029 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:02.728015 5939 scope.go:122] "RemoveContainer" containerID="029cc6d62e59f51f45b5dab11f4203fc62473cf7d62a515a170df932a32ade8f" Oct 31 01:16:02.735393 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:02.735377 5939 scope.go:122] "RemoveContainer" containerID="c410f6e0d5db442f23b005ac68ee576b50dd3dac8a2d00870c7d98a794001000" Oct 31 01:16:02.744583 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:02.744562 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl"] Oct 31 01:16:02.750796 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:02.750774 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-promote-v2-predictor-54ff8866fd-89rkl"] Oct 31 01:16:03.651792 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:03.651756 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" path="/var/lib/kubelet/pods/ad243c40-ad56-4462-b9c2-889d82e7c151/volumes" Oct 31 01:16:04.525769 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.525733 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx"] Oct 31 01:16:04.526154 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.526087 5939 cpu_manager.go:414] "containerMap: removing container" podUID="44cdfce8-192f-45a3-96d8-637027ad8a08" containerName="kserve-container" Oct 31 01:16:04.526154 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.526098 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44cdfce8-192f-45a3-96d8-637027ad8a08" containerName="kserve-container" Oct 31 01:16:04.526154 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.526117 5939 cpu_manager.go:414] "containerMap: removing container" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kserve-container" Oct 31 01:16:04.526154 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.526122 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kserve-container" Oct 31 01:16:04.526154 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.526133 5939 cpu_manager.go:414] "containerMap: removing container" podUID="44cdfce8-192f-45a3-96d8-637027ad8a08" containerName="kube-rbac-proxy" Oct 31 01:16:04.526154 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.526138 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44cdfce8-192f-45a3-96d8-637027ad8a08" containerName="kube-rbac-proxy" Oct 31 01:16:04.526154 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.526148 5939 cpu_manager.go:414] "containerMap: removing container" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="storage-initializer" Oct 31 01:16:04.526154 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.526152 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="storage-initializer" Oct 31 01:16:04.526397 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.526160 5939 cpu_manager.go:414] "containerMap: removing container" podUID="44cdfce8-192f-45a3-96d8-637027ad8a08" containerName="storage-initializer" Oct 31 01:16:04.526397 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.526165 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44cdfce8-192f-45a3-96d8-637027ad8a08" containerName="storage-initializer" Oct 31 01:16:04.526397 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.526176 5939 cpu_manager.go:414] "containerMap: removing container" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kube-rbac-proxy" Oct 31 01:16:04.526397 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.526180 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kube-rbac-proxy" Oct 31 01:16:04.526397 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.526232 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kserve-container" Oct 31 01:16:04.526397 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.526240 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="44cdfce8-192f-45a3-96d8-637027ad8a08" containerName="kube-rbac-proxy" Oct 31 01:16:04.526397 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.526248 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad243c40-ad56-4462-b9c2-889d82e7c151" containerName="kube-rbac-proxy" Oct 31 01:16:04.526397 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.526255 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="44cdfce8-192f-45a3-96d8-637027ad8a08" containerName="kserve-container" Oct 31 01:16:04.531021 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.531003 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:04.537704 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.537427 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-rollback-predictor-serving-cert\"" Oct 31 01:16:04.538307 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.538284 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\"" Oct 31 01:16:04.538986 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.538970 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\"" Oct 31 01:16:04.539063 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.539038 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\"" Oct 31 01:16:04.539063 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.539037 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 01:16:04.542596 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.542574 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx"] Oct 31 01:16:04.621864 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.621827 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-59c66949f9-mz8bx\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:04.622026 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.621949 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-59c66949f9-mz8bx\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:04.622026 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.621988 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht6nm\" (UniqueName: \"kubernetes.io/projected/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kube-api-access-ht6nm\") pod \"isvc-canary-rollback-predictor-59c66949f9-mz8bx\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:04.622138 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.622039 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d081dc8-4bff-43c9-b9e5-4f2e37737951-proxy-tls\") pod \"isvc-canary-rollback-predictor-59c66949f9-mz8bx\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:04.622138 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.622068 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d081dc8-4bff-43c9-b9e5-4f2e37737951-cabundle-cert\") pod \"isvc-canary-rollback-predictor-59c66949f9-mz8bx\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:04.635920 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.635894 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\"" Oct 31 01:16:04.636019 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.635953 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\"" Oct 31 01:16:04.723550 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.723519 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-59c66949f9-mz8bx\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:04.723742 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.723560 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht6nm\" (UniqueName: \"kubernetes.io/projected/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kube-api-access-ht6nm\") pod \"isvc-canary-rollback-predictor-59c66949f9-mz8bx\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:04.723742 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.723585 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d081dc8-4bff-43c9-b9e5-4f2e37737951-proxy-tls\") pod \"isvc-canary-rollback-predictor-59c66949f9-mz8bx\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:04.723742 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.723610 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d081dc8-4bff-43c9-b9e5-4f2e37737951-cabundle-cert\") pod \"isvc-canary-rollback-predictor-59c66949f9-mz8bx\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:04.723742 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.723656 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-59c66949f9-mz8bx\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:04.724114 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.724088 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-59c66949f9-mz8bx\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:04.724394 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.724371 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d081dc8-4bff-43c9-b9e5-4f2e37737951-cabundle-cert\") pod \"isvc-canary-rollback-predictor-59c66949f9-mz8bx\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:04.724533 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.724433 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-59c66949f9-mz8bx\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:04.726278 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.726256 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d081dc8-4bff-43c9-b9e5-4f2e37737951-proxy-tls\") pod \"isvc-canary-rollback-predictor-59c66949f9-mz8bx\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:04.737840 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.737819 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht6nm\" (UniqueName: \"kubernetes.io/projected/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kube-api-access-ht6nm\") pod \"isvc-canary-rollback-predictor-59c66949f9-mz8bx\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:04.848616 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:04.848534 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:05.129092 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:05.129066 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx"] Oct 31 01:16:05.132623 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:16:05.132592 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d081dc8_4bff_43c9_b9e5_4f2e37737951.slice/crio-a1ac2b56faeec4b07790a05cc0c817b7691bfe329921e72dc275a51403677549 WatchSource:0}: Error finding container a1ac2b56faeec4b07790a05cc0c817b7691bfe329921e72dc275a51403677549: Status 404 returned error can't find the container with id a1ac2b56faeec4b07790a05cc0c817b7691bfe329921e72dc275a51403677549 Oct 31 01:16:05.733546 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:05.733510 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" event={"ID":"5d081dc8-4bff-43c9-b9e5-4f2e37737951","Type":"ContainerStarted","Data":"e9d3b5ab836c3c6a754dff269b595f55cc5763670b2c68e1f09ed4e58730e4b5"} Oct 31 01:16:05.734057 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:05.733571 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" event={"ID":"5d081dc8-4bff-43c9-b9e5-4f2e37737951","Type":"ContainerStarted","Data":"a1ac2b56faeec4b07790a05cc0c817b7691bfe329921e72dc275a51403677549"} Oct 31 01:16:09.757332 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:09.757297 5939 generic.go:356] "Generic (PLEG): container finished" podID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerID="e9d3b5ab836c3c6a754dff269b595f55cc5763670b2c68e1f09ed4e58730e4b5" exitCode=0 Oct 31 01:16:09.757732 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:09.757353 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" event={"ID":"5d081dc8-4bff-43c9-b9e5-4f2e37737951","Type":"ContainerDied","Data":"e9d3b5ab836c3c6a754dff269b595f55cc5763670b2c68e1f09ed4e58730e4b5"} Oct 31 01:16:10.764179 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:10.764142 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" event={"ID":"5d081dc8-4bff-43c9-b9e5-4f2e37737951","Type":"ContainerStarted","Data":"59d23fc0cf27be87a92aae5e3e6c146a80ee0032f40683f84e90cc4abb6409df"} Oct 31 01:16:10.764720 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:10.764214 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" event={"ID":"5d081dc8-4bff-43c9-b9e5-4f2e37737951","Type":"ContainerStarted","Data":"77b221937d45ceff7066a6f12373677c1bbc7e64dc26bf5d97a9c39d1215b8aa"} Oct 31 01:16:10.764720 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:10.764434 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:10.792453 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:10.792404 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" podStartSLOduration=2.769288632 podStartE2EDuration="6.792390531s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.023101899s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:10.791692181 +0000 UTC m=+2783.280967359" watchObservedRunningTime="2026-10-31 01:16:10.792390531 +0000 UTC m=+2783.281665709" Oct 31 01:16:11.768169 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:11.768133 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:11.769499 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:11.769470 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused" Oct 31 01:16:12.771616 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:12.771579 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused" Oct 31 01:16:17.776196 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:17.776168 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:16:17.776729 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:17.776690 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused" Oct 31 01:16:27.777242 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:27.777203 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused" Oct 31 01:16:30.509344 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:16:30.509260 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:16:37.777064 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:37.777026 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused" Oct 31 01:16:47.777161 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:47.777127 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused" Oct 31 01:16:57.777437 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:16:57.777389 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused" Oct 31 01:17:00.508975 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:17:00.508940 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:17:07.777434 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:17:07.777389 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused" Oct 31 01:17:17.776914 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:17:17.776873 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused" Oct 31 01:17:22.648105 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:17:22.648056 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused" Oct 31 01:17:30.508971 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:17:30.508931 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:17:32.648866 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:17:32.648837 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:18:00.509421 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:18:00.509325 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:18:30.510661 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:18:30.510623 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:19:00.509366 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:19:00.509328 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:19:04.887624 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:04.887592 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx"] Oct 31 01:19:04.888059 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:04.887848 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kserve-container" containerID="cri-o://77b221937d45ceff7066a6f12373677c1bbc7e64dc26bf5d97a9c39d1215b8aa" gracePeriod=30 Oct 31 01:19:04.888059 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:04.887877 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kube-rbac-proxy" containerID="cri-o://59d23fc0cf27be87a92aae5e3e6c146a80ee0032f40683f84e90cc4abb6409df" gracePeriod=30 Oct 31 01:19:05.457267 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:05.457230 5939 generic.go:356] "Generic (PLEG): container finished" podID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerID="59d23fc0cf27be87a92aae5e3e6c146a80ee0032f40683f84e90cc4abb6409df" exitCode=0 Oct 31 01:19:05.457453 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:05.457299 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" event={"ID":"5d081dc8-4bff-43c9-b9e5-4f2e37737951","Type":"ContainerDied","Data":"59d23fc0cf27be87a92aae5e3e6c146a80ee0032f40683f84e90cc4abb6409df"} Oct 31 01:19:07.771928 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:07.771885 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused" Oct 31 01:19:10.125546 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.125520 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:19:10.272446 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.272362 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d081dc8-4bff-43c9-b9e5-4f2e37737951-proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d081dc8-4bff-43c9-b9e5-4f2e37737951-proxy-tls\") pod \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " Oct 31 01:19:10.272446 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.272401 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d081dc8-4bff-43c9-b9e5-4f2e37737951-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d081dc8-4bff-43c9-b9e5-4f2e37737951-cabundle-cert\") pod \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " Oct 31 01:19:10.272688 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.272463 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kserve-provision-location\") pod \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " Oct 31 01:19:10.272688 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.272514 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kube-rbac-proxy-sar-config\") pod \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " Oct 31 01:19:10.272688 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.272606 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kube-api-access-ht6nm\" (UniqueName: \"kubernetes.io/projected/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kube-api-access-ht6nm\") pod \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\" (UID: \"5d081dc8-4bff-43c9-b9e5-4f2e37737951\") " Oct 31 01:19:10.272868 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.272824 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kserve-provision-location" pod "5d081dc8-4bff-43c9-b9e5-4f2e37737951" (UID: "5d081dc8-4bff-43c9-b9e5-4f2e37737951"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:10.272929 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.272897 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:10.272929 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.272897 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d081dc8-4bff-43c9-b9e5-4f2e37737951-cabundle-cert" pod "5d081dc8-4bff-43c9-b9e5-4f2e37737951" (UID: "5d081dc8-4bff-43c9-b9e5-4f2e37737951"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:10.272929 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.272918 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kube-rbac-proxy-sar-config" pod "5d081dc8-4bff-43c9-b9e5-4f2e37737951" (UID: "5d081dc8-4bff-43c9-b9e5-4f2e37737951"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:10.274588 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.274562 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d081dc8-4bff-43c9-b9e5-4f2e37737951-proxy-tls" pod "5d081dc8-4bff-43c9-b9e5-4f2e37737951" (UID: "5d081dc8-4bff-43c9-b9e5-4f2e37737951"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:10.274822 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.274660 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kube-api-access-ht6nm" pod "5d081dc8-4bff-43c9-b9e5-4f2e37737951" (UID: "5d081dc8-4bff-43c9-b9e5-4f2e37737951"). InnerVolumeSpecName "kube-api-access-ht6nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:10.373530 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.373500 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d081dc8-4bff-43c9-b9e5-4f2e37737951-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:10.373530 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.373529 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:10.375485 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.373540 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht6nm\" (UniqueName: \"kubernetes.io/projected/5d081dc8-4bff-43c9-b9e5-4f2e37737951-kube-api-access-ht6nm\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:10.375485 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.373549 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d081dc8-4bff-43c9-b9e5-4f2e37737951-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:10.478086 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.478050 5939 generic.go:356] "Generic (PLEG): container finished" podID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerID="77b221937d45ceff7066a6f12373677c1bbc7e64dc26bf5d97a9c39d1215b8aa" exitCode=0 Oct 31 01:19:10.478298 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.478118 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" Oct 31 01:19:10.478298 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.478129 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" event={"ID":"5d081dc8-4bff-43c9-b9e5-4f2e37737951","Type":"ContainerDied","Data":"77b221937d45ceff7066a6f12373677c1bbc7e64dc26bf5d97a9c39d1215b8aa"} Oct 31 01:19:10.478298 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.478191 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx" event={"ID":"5d081dc8-4bff-43c9-b9e5-4f2e37737951","Type":"ContainerDied","Data":"a1ac2b56faeec4b07790a05cc0c817b7691bfe329921e72dc275a51403677549"} Oct 31 01:19:10.478298 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.478217 5939 scope.go:122] "RemoveContainer" containerID="59d23fc0cf27be87a92aae5e3e6c146a80ee0032f40683f84e90cc4abb6409df" Oct 31 01:19:10.488076 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.488058 5939 scope.go:122] "RemoveContainer" containerID="77b221937d45ceff7066a6f12373677c1bbc7e64dc26bf5d97a9c39d1215b8aa" Oct 31 01:19:10.495842 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.495823 5939 scope.go:122] "RemoveContainer" containerID="e9d3b5ab836c3c6a754dff269b595f55cc5763670b2c68e1f09ed4e58730e4b5" Oct 31 01:19:10.503933 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.503906 5939 scope.go:122] "RemoveContainer" containerID="59d23fc0cf27be87a92aae5e3e6c146a80ee0032f40683f84e90cc4abb6409df" Oct 31 01:19:10.504230 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:19:10.504211 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59d23fc0cf27be87a92aae5e3e6c146a80ee0032f40683f84e90cc4abb6409df\": container with ID starting with 59d23fc0cf27be87a92aae5e3e6c146a80ee0032f40683f84e90cc4abb6409df not found: ID does not exist" containerID="59d23fc0cf27be87a92aae5e3e6c146a80ee0032f40683f84e90cc4abb6409df" Oct 31 01:19:10.504326 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.504241 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59d23fc0cf27be87a92aae5e3e6c146a80ee0032f40683f84e90cc4abb6409df"} err="failed to get container status \"59d23fc0cf27be87a92aae5e3e6c146a80ee0032f40683f84e90cc4abb6409df\": rpc error: code = NotFound desc = could not find container \"59d23fc0cf27be87a92aae5e3e6c146a80ee0032f40683f84e90cc4abb6409df\": container with ID starting with 59d23fc0cf27be87a92aae5e3e6c146a80ee0032f40683f84e90cc4abb6409df not found: ID does not exist" Oct 31 01:19:10.504326 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.504264 5939 scope.go:122] "RemoveContainer" containerID="77b221937d45ceff7066a6f12373677c1bbc7e64dc26bf5d97a9c39d1215b8aa" Oct 31 01:19:10.504547 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:19:10.504531 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b221937d45ceff7066a6f12373677c1bbc7e64dc26bf5d97a9c39d1215b8aa\": container with ID starting with 77b221937d45ceff7066a6f12373677c1bbc7e64dc26bf5d97a9c39d1215b8aa not found: ID does not exist" containerID="77b221937d45ceff7066a6f12373677c1bbc7e64dc26bf5d97a9c39d1215b8aa" Oct 31 01:19:10.504596 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.504551 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b221937d45ceff7066a6f12373677c1bbc7e64dc26bf5d97a9c39d1215b8aa"} err="failed to get container status \"77b221937d45ceff7066a6f12373677c1bbc7e64dc26bf5d97a9c39d1215b8aa\": rpc error: code = NotFound desc = could not find container \"77b221937d45ceff7066a6f12373677c1bbc7e64dc26bf5d97a9c39d1215b8aa\": container with ID starting with 77b221937d45ceff7066a6f12373677c1bbc7e64dc26bf5d97a9c39d1215b8aa not found: ID does not exist" Oct 31 01:19:10.504596 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.504563 5939 scope.go:122] "RemoveContainer" containerID="e9d3b5ab836c3c6a754dff269b595f55cc5763670b2c68e1f09ed4e58730e4b5" Oct 31 01:19:10.504802 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.504725 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx"] Oct 31 01:19:10.504928 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:19:10.504906 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d3b5ab836c3c6a754dff269b595f55cc5763670b2c68e1f09ed4e58730e4b5\": container with ID starting with e9d3b5ab836c3c6a754dff269b595f55cc5763670b2c68e1f09ed4e58730e4b5 not found: ID does not exist" containerID="e9d3b5ab836c3c6a754dff269b595f55cc5763670b2c68e1f09ed4e58730e4b5" Oct 31 01:19:10.504984 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.504938 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d3b5ab836c3c6a754dff269b595f55cc5763670b2c68e1f09ed4e58730e4b5"} err="failed to get container status \"e9d3b5ab836c3c6a754dff269b595f55cc5763670b2c68e1f09ed4e58730e4b5\": rpc error: code = NotFound desc = could not find container \"e9d3b5ab836c3c6a754dff269b595f55cc5763670b2c68e1f09ed4e58730e4b5\": container with ID starting with e9d3b5ab836c3c6a754dff269b595f55cc5763670b2c68e1f09ed4e58730e4b5 not found: ID does not exist" Oct 31 01:19:10.511188 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:10.511165 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-rollback-predictor-59c66949f9-mz8bx"] Oct 31 01:19:11.542986 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.542956 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw"] Oct 31 01:19:11.543380 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.543282 5939 cpu_manager.go:414] "containerMap: removing container" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kube-rbac-proxy" Oct 31 01:19:11.543380 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.543292 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kube-rbac-proxy" Oct 31 01:19:11.543380 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.543307 5939 cpu_manager.go:414] "containerMap: removing container" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kserve-container" Oct 31 01:19:11.543380 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.543312 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kserve-container" Oct 31 01:19:11.543380 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.543338 5939 cpu_manager.go:414] "containerMap: removing container" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="storage-initializer" Oct 31 01:19:11.543380 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.543343 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="storage-initializer" Oct 31 01:19:11.543571 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.543392 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kube-rbac-proxy" Oct 31 01:19:11.543571 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.543401 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" containerName="kserve-container" Oct 31 01:19:11.547862 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.547844 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:11.560215 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.560185 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw"] Oct 31 01:19:11.599372 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.599342 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 01:19:11.599574 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.599373 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\"" Oct 31 01:19:11.599574 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.599394 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\"" Oct 31 01:19:11.599574 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.599401 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\"" Oct 31 01:19:11.599574 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.599499 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\"" Oct 31 01:19:11.599574 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.599561 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-canary-stop-predictor-serving-cert\"" Oct 31 01:19:11.599574 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.599534 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\"" Oct 31 01:19:11.652626 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.652591 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d081dc8-4bff-43c9-b9e5-4f2e37737951" path="/var/lib/kubelet/pods/5d081dc8-4bff-43c9-b9e5-4f2e37737951/volumes" Oct 31 01:19:11.684899 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.684872 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e74220e-fc21-4729-8118-2fea744a4424-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-8brsw\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:11.685041 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.684944 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e74220e-fc21-4729-8118-2fea744a4424-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-8brsw\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:11.685041 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.685012 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e74220e-fc21-4729-8118-2fea744a4424-proxy-tls\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-8brsw\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:11.685153 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.685044 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwtrx\" (UniqueName: \"kubernetes.io/projected/1e74220e-fc21-4729-8118-2fea744a4424-kube-api-access-xwtrx\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-8brsw\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:11.685153 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.685082 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e74220e-fc21-4729-8118-2fea744a4424-cabundle-cert\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-8brsw\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:11.786571 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.786520 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e74220e-fc21-4729-8118-2fea744a4424-cabundle-cert\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-8brsw\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:11.786805 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.786592 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e74220e-fc21-4729-8118-2fea744a4424-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-8brsw\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:11.786805 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.786692 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e74220e-fc21-4729-8118-2fea744a4424-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-8brsw\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:11.786805 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.786717 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e74220e-fc21-4729-8118-2fea744a4424-proxy-tls\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-8brsw\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:11.786805 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.786747 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwtrx\" (UniqueName: \"kubernetes.io/projected/1e74220e-fc21-4729-8118-2fea744a4424-kube-api-access-xwtrx\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-8brsw\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:11.787125 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.787096 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e74220e-fc21-4729-8118-2fea744a4424-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-8brsw\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:11.787393 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.787371 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e74220e-fc21-4729-8118-2fea744a4424-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-8brsw\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:11.787431 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.787402 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e74220e-fc21-4729-8118-2fea744a4424-cabundle-cert\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-8brsw\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:11.789192 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.789166 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e74220e-fc21-4729-8118-2fea744a4424-proxy-tls\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-8brsw\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:11.801575 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.801521 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwtrx\" (UniqueName: \"kubernetes.io/projected/1e74220e-fc21-4729-8118-2fea744a4424-kube-api-access-xwtrx\") pod \"isvc-canary-stop-predictor-59cc4ccb6c-8brsw\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:11.865280 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:11.865245 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:12.147422 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:12.147387 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw"] Oct 31 01:19:12.150746 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:19:12.150715 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e74220e_fc21_4729_8118_2fea744a4424.slice/crio-95ad06629214b2887adc775a440c766484aea2567acfc4a7113be410a5b47356 WatchSource:0}: Error finding container 95ad06629214b2887adc775a440c766484aea2567acfc4a7113be410a5b47356: Status 404 returned error can't find the container with id 95ad06629214b2887adc775a440c766484aea2567acfc4a7113be410a5b47356 Oct 31 01:19:12.489826 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:12.489734 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" event={"ID":"1e74220e-fc21-4729-8118-2fea744a4424","Type":"ContainerStarted","Data":"ce9b3775d89e156b1e86a2f394c299aeb2d01a3ea60ecdc5d3dcbaf259fe06ea"} Oct 31 01:19:12.489826 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:12.489789 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" event={"ID":"1e74220e-fc21-4729-8118-2fea744a4424","Type":"ContainerStarted","Data":"95ad06629214b2887adc775a440c766484aea2567acfc4a7113be410a5b47356"} Oct 31 01:19:16.508768 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:16.508657 5939 generic.go:356] "Generic (PLEG): container finished" podID="1e74220e-fc21-4729-8118-2fea744a4424" containerID="ce9b3775d89e156b1e86a2f394c299aeb2d01a3ea60ecdc5d3dcbaf259fe06ea" exitCode=0 Oct 31 01:19:16.508768 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:16.508733 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" event={"ID":"1e74220e-fc21-4729-8118-2fea744a4424","Type":"ContainerDied","Data":"ce9b3775d89e156b1e86a2f394c299aeb2d01a3ea60ecdc5d3dcbaf259fe06ea"} Oct 31 01:19:17.515823 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:17.515777 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" event={"ID":"1e74220e-fc21-4729-8118-2fea744a4424","Type":"ContainerStarted","Data":"287326fb321b4657ecc89fb4f6a05acfa228d2b81e0c9b89b8b240503ac10906"} Oct 31 01:19:17.516250 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:17.515868 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" event={"ID":"1e74220e-fc21-4729-8118-2fea744a4424","Type":"ContainerStarted","Data":"c8bb6862109db83d77ceb59bdec9ac90818af3776630561cb6be9f16fea7184f"} Oct 31 01:19:17.516250 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:17.516094 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:17.545475 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:17.545426 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" podStartSLOduration=2.644087297 podStartE2EDuration="6.545412021s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.901324724s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:17.543292979 +0000 UTC m=+2970.032568149" watchObservedRunningTime="2026-10-31 01:19:17.545412021 +0000 UTC m=+2970.034687199" Oct 31 01:19:18.521365 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:18.521330 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:18.522579 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:18.522552 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused" Oct 31 01:19:19.525488 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:19.525451 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused" Oct 31 01:19:24.530609 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:24.530580 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:19:24.531264 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:24.531237 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused" Oct 31 01:19:30.508585 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:19:30.508510 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:19:34.531955 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:34.531913 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused" Oct 31 01:19:44.531178 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:44.531137 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused" Oct 31 01:19:47.999603 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:47.999574 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 01:19:48.001064 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:48.001044 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 01:19:54.531648 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:19:54.531610 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused" Oct 31 01:20:00.509719 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:20:00.509661 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:20:04.531421 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:04.531380 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused" Oct 31 01:20:14.531908 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:14.531867 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused" Oct 31 01:20:24.531726 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:24.531653 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused" Oct 31 01:20:30.509572 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:20:30.509534 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:20:33.648364 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:33.648302 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused" Oct 31 01:20:38.407387 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:38.407352 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw"] Oct 31 01:20:38.407871 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:38.407558 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kserve-container" containerID="cri-o://c8bb6862109db83d77ceb59bdec9ac90818af3776630561cb6be9f16fea7184f" gracePeriod=30 Oct 31 01:20:38.407871 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:38.407624 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kube-rbac-proxy" containerID="cri-o://287326fb321b4657ecc89fb4f6a05acfa228d2b81e0c9b89b8b240503ac10906" gracePeriod=30 Oct 31 01:20:38.408656 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:38.408637 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:20:38.862630 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:38.862599 5939 generic.go:356] "Generic (PLEG): container finished" podID="1e74220e-fc21-4729-8118-2fea744a4424" containerID="287326fb321b4657ecc89fb4f6a05acfa228d2b81e0c9b89b8b240503ac10906" exitCode=0 Oct 31 01:20:38.862817 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:38.862694 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" event={"ID":"1e74220e-fc21-4729-8118-2fea744a4424","Type":"ContainerDied","Data":"287326fb321b4657ecc89fb4f6a05acfa228d2b81e0c9b89b8b240503ac10906"} Oct 31 01:20:39.525950 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:39.525907 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused" Oct 31 01:20:43.651712 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.651683 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:20:43.799203 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.799110 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e74220e-fc21-4729-8118-2fea744a4424-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e74220e-fc21-4729-8118-2fea744a4424-kserve-provision-location\") pod \"1e74220e-fc21-4729-8118-2fea744a4424\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " Oct 31 01:20:43.799392 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.799221 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e74220e-fc21-4729-8118-2fea744a4424-kube-api-access-xwtrx\" (UniqueName: \"kubernetes.io/projected/1e74220e-fc21-4729-8118-2fea744a4424-kube-api-access-xwtrx\") pod \"1e74220e-fc21-4729-8118-2fea744a4424\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " Oct 31 01:20:43.799392 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.799246 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e74220e-fc21-4729-8118-2fea744a4424-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e74220e-fc21-4729-8118-2fea744a4424-proxy-tls\") pod \"1e74220e-fc21-4729-8118-2fea744a4424\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " Oct 31 01:20:43.799392 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.799277 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e74220e-fc21-4729-8118-2fea744a4424-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e74220e-fc21-4729-8118-2fea744a4424-kube-rbac-proxy-sar-config\") pod \"1e74220e-fc21-4729-8118-2fea744a4424\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " Oct 31 01:20:43.799392 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.799299 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e74220e-fc21-4729-8118-2fea744a4424-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e74220e-fc21-4729-8118-2fea744a4424-cabundle-cert\") pod \"1e74220e-fc21-4729-8118-2fea744a4424\" (UID: \"1e74220e-fc21-4729-8118-2fea744a4424\") " Oct 31 01:20:43.799618 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.799513 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e74220e-fc21-4729-8118-2fea744a4424-kserve-provision-location" pod "1e74220e-fc21-4729-8118-2fea744a4424" (UID: "1e74220e-fc21-4729-8118-2fea744a4424"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:43.799663 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.799631 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e74220e-fc21-4729-8118-2fea744a4424-kube-rbac-proxy-sar-config" pod "1e74220e-fc21-4729-8118-2fea744a4424" (UID: "1e74220e-fc21-4729-8118-2fea744a4424"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:43.799740 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.799719 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e74220e-fc21-4729-8118-2fea744a4424-cabundle-cert" pod "1e74220e-fc21-4729-8118-2fea744a4424" (UID: "1e74220e-fc21-4729-8118-2fea744a4424"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:43.801410 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.801388 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e74220e-fc21-4729-8118-2fea744a4424-proxy-tls" pod "1e74220e-fc21-4729-8118-2fea744a4424" (UID: "1e74220e-fc21-4729-8118-2fea744a4424"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:43.801525 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.801472 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e74220e-fc21-4729-8118-2fea744a4424-kube-api-access-xwtrx" pod "1e74220e-fc21-4729-8118-2fea744a4424" (UID: "1e74220e-fc21-4729-8118-2fea744a4424"). InnerVolumeSpecName "kube-api-access-xwtrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:43.885001 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.884967 5939 generic.go:356] "Generic (PLEG): container finished" podID="1e74220e-fc21-4729-8118-2fea744a4424" containerID="c8bb6862109db83d77ceb59bdec9ac90818af3776630561cb6be9f16fea7184f" exitCode=0 Oct 31 01:20:43.885157 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.885024 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" Oct 31 01:20:43.885157 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.885059 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" event={"ID":"1e74220e-fc21-4729-8118-2fea744a4424","Type":"ContainerDied","Data":"c8bb6862109db83d77ceb59bdec9ac90818af3776630561cb6be9f16fea7184f"} Oct 31 01:20:43.885157 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.885134 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw" event={"ID":"1e74220e-fc21-4729-8118-2fea744a4424","Type":"ContainerDied","Data":"95ad06629214b2887adc775a440c766484aea2567acfc4a7113be410a5b47356"} Oct 31 01:20:43.885289 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.885172 5939 scope.go:122] "RemoveContainer" containerID="287326fb321b4657ecc89fb4f6a05acfa228d2b81e0c9b89b8b240503ac10906" Oct 31 01:20:43.894483 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.894459 5939 scope.go:122] "RemoveContainer" containerID="c8bb6862109db83d77ceb59bdec9ac90818af3776630561cb6be9f16fea7184f" Oct 31 01:20:43.899794 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.899772 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwtrx\" (UniqueName: \"kubernetes.io/projected/1e74220e-fc21-4729-8118-2fea744a4424-kube-api-access-xwtrx\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:43.899794 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.899792 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e74220e-fc21-4729-8118-2fea744a4424-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:43.899956 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.899802 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e74220e-fc21-4729-8118-2fea744a4424-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:43.899956 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.899812 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e74220e-fc21-4729-8118-2fea744a4424-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:43.899956 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.899821 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e74220e-fc21-4729-8118-2fea744a4424-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:43.902020 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.902006 5939 scope.go:122] "RemoveContainer" containerID="ce9b3775d89e156b1e86a2f394c299aeb2d01a3ea60ecdc5d3dcbaf259fe06ea" Oct 31 01:20:43.909429 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.909412 5939 scope.go:122] "RemoveContainer" containerID="287326fb321b4657ecc89fb4f6a05acfa228d2b81e0c9b89b8b240503ac10906" Oct 31 01:20:43.909700 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:20:43.909683 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287326fb321b4657ecc89fb4f6a05acfa228d2b81e0c9b89b8b240503ac10906\": container with ID starting with 287326fb321b4657ecc89fb4f6a05acfa228d2b81e0c9b89b8b240503ac10906 not found: ID does not exist" containerID="287326fb321b4657ecc89fb4f6a05acfa228d2b81e0c9b89b8b240503ac10906" Oct 31 01:20:43.909761 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.909710 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287326fb321b4657ecc89fb4f6a05acfa228d2b81e0c9b89b8b240503ac10906"} err="failed to get container status \"287326fb321b4657ecc89fb4f6a05acfa228d2b81e0c9b89b8b240503ac10906\": rpc error: code = NotFound desc = could not find container \"287326fb321b4657ecc89fb4f6a05acfa228d2b81e0c9b89b8b240503ac10906\": container with ID starting with 287326fb321b4657ecc89fb4f6a05acfa228d2b81e0c9b89b8b240503ac10906 not found: ID does not exist" Oct 31 01:20:43.909761 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.909729 5939 scope.go:122] "RemoveContainer" containerID="c8bb6862109db83d77ceb59bdec9ac90818af3776630561cb6be9f16fea7184f" Oct 31 01:20:43.909986 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:20:43.909966 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8bb6862109db83d77ceb59bdec9ac90818af3776630561cb6be9f16fea7184f\": container with ID starting with c8bb6862109db83d77ceb59bdec9ac90818af3776630561cb6be9f16fea7184f not found: ID does not exist" containerID="c8bb6862109db83d77ceb59bdec9ac90818af3776630561cb6be9f16fea7184f" Oct 31 01:20:43.910063 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.909998 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8bb6862109db83d77ceb59bdec9ac90818af3776630561cb6be9f16fea7184f"} err="failed to get container status \"c8bb6862109db83d77ceb59bdec9ac90818af3776630561cb6be9f16fea7184f\": rpc error: code = NotFound desc = could not find container \"c8bb6862109db83d77ceb59bdec9ac90818af3776630561cb6be9f16fea7184f\": container with ID starting with c8bb6862109db83d77ceb59bdec9ac90818af3776630561cb6be9f16fea7184f not found: ID does not exist" Oct 31 01:20:43.910063 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.910022 5939 scope.go:122] "RemoveContainer" containerID="ce9b3775d89e156b1e86a2f394c299aeb2d01a3ea60ecdc5d3dcbaf259fe06ea" Oct 31 01:20:43.910292 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:20:43.910275 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce9b3775d89e156b1e86a2f394c299aeb2d01a3ea60ecdc5d3dcbaf259fe06ea\": container with ID starting with ce9b3775d89e156b1e86a2f394c299aeb2d01a3ea60ecdc5d3dcbaf259fe06ea not found: ID does not exist" containerID="ce9b3775d89e156b1e86a2f394c299aeb2d01a3ea60ecdc5d3dcbaf259fe06ea" Oct 31 01:20:43.910328 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.910297 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce9b3775d89e156b1e86a2f394c299aeb2d01a3ea60ecdc5d3dcbaf259fe06ea"} err="failed to get container status \"ce9b3775d89e156b1e86a2f394c299aeb2d01a3ea60ecdc5d3dcbaf259fe06ea\": rpc error: code = NotFound desc = could not find container \"ce9b3775d89e156b1e86a2f394c299aeb2d01a3ea60ecdc5d3dcbaf259fe06ea\": container with ID starting with ce9b3775d89e156b1e86a2f394c299aeb2d01a3ea60ecdc5d3dcbaf259fe06ea not found: ID does not exist" Oct 31 01:20:43.912641 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.912617 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw"] Oct 31 01:20:43.919127 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:43.919105 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-canary-stop-predictor-59cc4ccb6c-8brsw"] Oct 31 01:20:45.651865 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:45.651826 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e74220e-fc21-4729-8118-2fea744a4424" path="/var/lib/kubelet/pods/1e74220e-fc21-4729-8118-2fea744a4424/volumes" Oct 31 01:20:46.226360 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.226328 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc"] Oct 31 01:20:46.226687 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.226658 5939 cpu_manager.go:414] "containerMap: removing container" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="storage-initializer" Oct 31 01:20:46.226687 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.226684 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="storage-initializer" Oct 31 01:20:46.226779 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.226702 5939 cpu_manager.go:414] "containerMap: removing container" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kserve-container" Oct 31 01:20:46.226779 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.226710 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kserve-container" Oct 31 01:20:46.226779 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.226731 5939 cpu_manager.go:414] "containerMap: removing container" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kube-rbac-proxy" Oct 31 01:20:46.226779 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.226738 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kube-rbac-proxy" Oct 31 01:20:46.226911 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.226816 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kserve-container" Oct 31 01:20:46.226911 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.226829 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e74220e-fc21-4729-8118-2fea744a4424" containerName="kube-rbac-proxy" Oct 31 01:20:46.231621 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.231601 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" Oct 31 01:20:46.238805 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.238779 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\"" Oct 31 01:20:46.240517 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.240482 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"openshift-service-ca.crt\"" Oct 31 01:20:46.240517 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.240491 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-1c69aea164\"/\"kube-root-ca.crt\"" Oct 31 01:20:46.244376 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.244351 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc"] Oct 31 01:20:46.256467 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.256439 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"storage-config\"" Oct 31 01:20:46.256610 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.256481 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"default-dockercfg-dhqsr\"" Oct 31 01:20:46.256610 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.256537 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-1c69aea164\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\"" Oct 31 01:20:46.321099 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.321063 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5a98608-6243-485a-8e03-070c72208db6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc\" (UID: \"f5a98608-6243-485a-8e03-070c72208db6\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" Oct 31 01:20:46.321286 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.321117 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5a98608-6243-485a-8e03-070c72208db6-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc\" (UID: \"f5a98608-6243-485a-8e03-070c72208db6\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" Oct 31 01:20:46.321286 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.321255 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6m2s\" (UniqueName: \"kubernetes.io/projected/f5a98608-6243-485a-8e03-070c72208db6-kube-api-access-k6m2s\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc\" (UID: \"f5a98608-6243-485a-8e03-070c72208db6\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" Oct 31 01:20:46.321358 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.321290 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5a98608-6243-485a-8e03-070c72208db6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc\" (UID: \"f5a98608-6243-485a-8e03-070c72208db6\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" Oct 31 01:20:46.422346 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.422311 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5a98608-6243-485a-8e03-070c72208db6-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc\" (UID: \"f5a98608-6243-485a-8e03-070c72208db6\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" Oct 31 01:20:46.422492 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.422399 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6m2s\" (UniqueName: \"kubernetes.io/projected/f5a98608-6243-485a-8e03-070c72208db6-kube-api-access-k6m2s\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc\" (UID: \"f5a98608-6243-485a-8e03-070c72208db6\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" Oct 31 01:20:46.422492 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.422434 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5a98608-6243-485a-8e03-070c72208db6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc\" (UID: \"f5a98608-6243-485a-8e03-070c72208db6\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" Oct 31 01:20:46.422492 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.422486 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5a98608-6243-485a-8e03-070c72208db6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc\" (UID: \"f5a98608-6243-485a-8e03-070c72208db6\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" Oct 31 01:20:46.422967 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.422944 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5a98608-6243-485a-8e03-070c72208db6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc\" (UID: \"f5a98608-6243-485a-8e03-070c72208db6\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" Oct 31 01:20:46.423199 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.423178 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5a98608-6243-485a-8e03-070c72208db6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc\" (UID: \"f5a98608-6243-485a-8e03-070c72208db6\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" Oct 31 01:20:46.424847 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.424829 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5a98608-6243-485a-8e03-070c72208db6-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc\" (UID: \"f5a98608-6243-485a-8e03-070c72208db6\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" Oct 31 01:20:46.436935 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.436909 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6m2s\" (UniqueName: \"kubernetes.io/projected/f5a98608-6243-485a-8e03-070c72208db6-kube-api-access-k6m2s\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc\" (UID: \"f5a98608-6243-485a-8e03-070c72208db6\") " pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" Oct 31 01:20:46.548952 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.548856 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" Oct 31 01:20:46.828482 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.828404 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc"] Oct 31 01:20:46.831938 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:20:46.831909 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a98608_6243_485a_8e03_070c72208db6.slice/crio-ad057041f0ffb75b9ded8a6f6a929c3fc29a18bc24230c24bf1ef3cfe747b4b6 WatchSource:0}: Error finding container ad057041f0ffb75b9ded8a6f6a929c3fc29a18bc24230c24bf1ef3cfe747b4b6: Status 404 returned error can't find the container with id ad057041f0ffb75b9ded8a6f6a929c3fc29a18bc24230c24bf1ef3cfe747b4b6 Oct 31 01:20:46.902030 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.901998 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" event={"ID":"f5a98608-6243-485a-8e03-070c72208db6","Type":"ContainerStarted","Data":"c2bf3e131f26c6536a82648847e7b985a18891a8d56f5b21a6afe6a587dd0b93"} Oct 31 01:20:46.902239 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:46.902059 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" event={"ID":"f5a98608-6243-485a-8e03-070c72208db6","Type":"ContainerStarted","Data":"ad057041f0ffb75b9ded8a6f6a929c3fc29a18bc24230c24bf1ef3cfe747b4b6"} Oct 31 01:20:49.915545 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:49.915514 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc_f5a98608-6243-485a-8e03-070c72208db6/storage-initializer/0.log" Oct 31 01:20:49.915967 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:49.915551 5939 generic.go:356] "Generic (PLEG): container finished" podID="f5a98608-6243-485a-8e03-070c72208db6" containerID="c2bf3e131f26c6536a82648847e7b985a18891a8d56f5b21a6afe6a587dd0b93" exitCode=1 Oct 31 01:20:49.915967 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:49.915594 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" event={"ID":"f5a98608-6243-485a-8e03-070c72208db6","Type":"ContainerDied","Data":"c2bf3e131f26c6536a82648847e7b985a18891a8d56f5b21a6afe6a587dd0b93"} Oct 31 01:20:50.921310 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:50.921279 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc_f5a98608-6243-485a-8e03-070c72208db6/storage-initializer/0.log" Oct 31 01:20:50.921770 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:50.921395 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" event={"ID":"f5a98608-6243-485a-8e03-070c72208db6","Type":"ContainerStarted","Data":"a3cb088a4853e1076cb1e0a521852f82c0687e229b0a988bfe8eb03a91ad2b38"} Oct 31 01:20:51.927065 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:51.927035 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc_f5a98608-6243-485a-8e03-070c72208db6/storage-initializer/1.log" Oct 31 01:20:51.927470 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:51.927409 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc_f5a98608-6243-485a-8e03-070c72208db6/storage-initializer/0.log" Oct 31 01:20:51.927470 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:51.927439 5939 generic.go:356] "Generic (PLEG): container finished" podID="f5a98608-6243-485a-8e03-070c72208db6" containerID="a3cb088a4853e1076cb1e0a521852f82c0687e229b0a988bfe8eb03a91ad2b38" exitCode=1 Oct 31 01:20:51.927562 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:51.927477 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" event={"ID":"f5a98608-6243-485a-8e03-070c72208db6","Type":"ContainerDied","Data":"a3cb088a4853e1076cb1e0a521852f82c0687e229b0a988bfe8eb03a91ad2b38"} Oct 31 01:20:51.927562 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:51.927528 5939 scope.go:122] "RemoveContainer" containerID="c2bf3e131f26c6536a82648847e7b985a18891a8d56f5b21a6afe6a587dd0b93" Oct 31 01:20:51.927951 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:51.927931 5939 scope.go:122] "RemoveContainer" containerID="c2bf3e131f26c6536a82648847e7b985a18891a8d56f5b21a6afe6a587dd0b93" Oct 31 01:20:51.938599 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:20:51.938574 5939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc_e2e-gw1-1c69aea164(f5a98608-6243-485a-8e03-070c72208db6)\"" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" podUID="f5a98608-6243-485a-8e03-070c72208db6" Oct 31 01:20:51.938768 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:20:51.938736 5939 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc_e2e-gw1-1c69aea164_f5a98608-6243-485a-8e03-070c72208db6_0 in pod sandbox ad057041f0ffb75b9ded8a6f6a929c3fc29a18bc24230c24bf1ef3cfe747b4b6 from index: no such id: 'c2bf3e131f26c6536a82648847e7b985a18891a8d56f5b21a6afe6a587dd0b93'" containerID="c2bf3e131f26c6536a82648847e7b985a18891a8d56f5b21a6afe6a587dd0b93" Oct 31 01:20:51.938847 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:51.938775 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2bf3e131f26c6536a82648847e7b985a18891a8d56f5b21a6afe6a587dd0b93"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc_e2e-gw1-1c69aea164_f5a98608-6243-485a-8e03-070c72208db6_0 in pod sandbox ad057041f0ffb75b9ded8a6f6a929c3fc29a18bc24230c24bf1ef3cfe747b4b6 from index: no such id: 'c2bf3e131f26c6536a82648847e7b985a18891a8d56f5b21a6afe6a587dd0b93'" Oct 31 01:20:52.933418 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:52.933389 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc_f5a98608-6243-485a-8e03-070c72208db6/storage-initializer/1.log" Oct 31 01:20:56.164407 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.164368 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc"] Oct 31 01:20:56.293783 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.293510 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc_f5a98608-6243-485a-8e03-070c72208db6/storage-initializer/1.log" Oct 31 01:20:56.297124 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.297097 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" Oct 31 01:20:56.410290 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.410251 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a98608-6243-485a-8e03-070c72208db6-kube-api-access-k6m2s\" (UniqueName: \"kubernetes.io/projected/f5a98608-6243-485a-8e03-070c72208db6-kube-api-access-k6m2s\") pod \"f5a98608-6243-485a-8e03-070c72208db6\" (UID: \"f5a98608-6243-485a-8e03-070c72208db6\") " Oct 31 01:20:56.410467 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.410337 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a98608-6243-485a-8e03-070c72208db6-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5a98608-6243-485a-8e03-070c72208db6-kserve-provision-location\") pod \"f5a98608-6243-485a-8e03-070c72208db6\" (UID: \"f5a98608-6243-485a-8e03-070c72208db6\") " Oct 31 01:20:56.410467 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.410369 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5a98608-6243-485a-8e03-070c72208db6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5a98608-6243-485a-8e03-070c72208db6-kube-rbac-proxy-sar-config\") pod \"f5a98608-6243-485a-8e03-070c72208db6\" (UID: \"f5a98608-6243-485a-8e03-070c72208db6\") " Oct 31 01:20:56.410467 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.410395 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5a98608-6243-485a-8e03-070c72208db6-proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5a98608-6243-485a-8e03-070c72208db6-proxy-tls\") pod \"f5a98608-6243-485a-8e03-070c72208db6\" (UID: \"f5a98608-6243-485a-8e03-070c72208db6\") " Oct 31 01:20:56.410722 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.410695 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a98608-6243-485a-8e03-070c72208db6-kserve-provision-location" pod "f5a98608-6243-485a-8e03-070c72208db6" (UID: "f5a98608-6243-485a-8e03-070c72208db6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:56.410805 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.410784 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a98608-6243-485a-8e03-070c72208db6-kube-rbac-proxy-sar-config" pod "f5a98608-6243-485a-8e03-070c72208db6" (UID: "f5a98608-6243-485a-8e03-070c72208db6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:56.412537 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.412508 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a98608-6243-485a-8e03-070c72208db6-kube-api-access-k6m2s" pod "f5a98608-6243-485a-8e03-070c72208db6" (UID: "f5a98608-6243-485a-8e03-070c72208db6"). InnerVolumeSpecName "kube-api-access-k6m2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:56.412655 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.412580 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a98608-6243-485a-8e03-070c72208db6-proxy-tls" pod "f5a98608-6243-485a-8e03-070c72208db6" (UID: "f5a98608-6243-485a-8e03-070c72208db6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:56.511915 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.511870 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5a98608-6243-485a-8e03-070c72208db6-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:56.511915 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.511908 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5a98608-6243-485a-8e03-070c72208db6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:56.512139 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.511949 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5a98608-6243-485a-8e03-070c72208db6-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:56.512139 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.511965 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6m2s\" (UniqueName: \"kubernetes.io/projected/f5a98608-6243-485a-8e03-070c72208db6-kube-api-access-k6m2s\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:56.952057 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.952027 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-1c69aea164_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc_f5a98608-6243-485a-8e03-070c72208db6/storage-initializer/1.log" Oct 31 01:20:56.952206 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.952091 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" event={"ID":"f5a98608-6243-485a-8e03-070c72208db6","Type":"ContainerDied","Data":"ad057041f0ffb75b9ded8a6f6a929c3fc29a18bc24230c24bf1ef3cfe747b4b6"} Oct 31 01:20:56.952206 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.952123 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc" Oct 31 01:20:56.952206 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.952145 5939 scope.go:122] "RemoveContainer" containerID="a3cb088a4853e1076cb1e0a521852f82c0687e229b0a988bfe8eb03a91ad2b38" Oct 31 01:20:56.997830 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:56.997797 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc"] Oct 31 01:20:57.005318 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:57.005290 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-1c69aea164/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd4f4bb-cmmsc"] Oct 31 01:20:57.651937 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:20:57.651906 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5a98608-6243-485a-8e03-070c72208db6" path="/var/lib/kubelet/pods/f5a98608-6243-485a-8e03-070c72208db6/volumes" Oct 31 01:21:00.509394 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:21:00.509319 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:21:30.509159 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:21:30.509123 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:22:00.508825 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:22:00.508791 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:22:30.510217 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:22:30.510123 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:22:53.738946 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.738913 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr"] Oct 31 01:22:53.739418 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.739254 5939 cpu_manager.go:414] "containerMap: removing container" podUID="f5a98608-6243-485a-8e03-070c72208db6" containerName="storage-initializer" Oct 31 01:22:53.739418 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.739268 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a98608-6243-485a-8e03-070c72208db6" containerName="storage-initializer" Oct 31 01:22:53.739418 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.739281 5939 cpu_manager.go:414] "containerMap: removing container" podUID="f5a98608-6243-485a-8e03-070c72208db6" containerName="storage-initializer" Oct 31 01:22:53.739418 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.739286 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a98608-6243-485a-8e03-070c72208db6" containerName="storage-initializer" Oct 31 01:22:53.739418 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.739342 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a98608-6243-485a-8e03-070c72208db6" containerName="storage-initializer" Oct 31 01:22:53.739418 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.739351 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a98608-6243-485a-8e03-070c72208db6" containerName="storage-initializer" Oct 31 01:22:53.742534 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.742512 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:22:53.751330 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.751310 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\"" Oct 31 01:22:53.751833 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.751807 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 01:22:53.751833 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.751807 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\"" Oct 31 01:22:53.751924 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.751808 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\"" Oct 31 01:22:53.751924 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.751818 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\"" Oct 31 01:22:53.751924 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.751870 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-lightgbm-runtime-predictor-serving-cert\"" Oct 31 01:22:53.752016 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.751929 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\"" Oct 31 01:22:53.755795 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.755775 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr"] Oct 31 01:22:53.776897 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.776872 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/551ab150-0945-44f7-89f2-f6614dd406db-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:22:53.777004 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.776907 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/551ab150-0945-44f7-89f2-f6614dd406db-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:22:53.777004 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.776974 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkwgh\" (UniqueName: \"kubernetes.io/projected/551ab150-0945-44f7-89f2-f6614dd406db-kube-api-access-hkwgh\") pod \"isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:22:53.777090 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.777048 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/551ab150-0945-44f7-89f2-f6614dd406db-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:22:53.777090 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.777079 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/551ab150-0945-44f7-89f2-f6614dd406db-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:22:53.878330 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.878298 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/551ab150-0945-44f7-89f2-f6614dd406db-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:22:53.878489 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.878338 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/551ab150-0945-44f7-89f2-f6614dd406db-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:22:53.878624 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.878597 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/551ab150-0945-44f7-89f2-f6614dd406db-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:22:53.878704 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.878685 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/551ab150-0945-44f7-89f2-f6614dd406db-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:22:53.878756 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.878724 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkwgh\" (UniqueName: \"kubernetes.io/projected/551ab150-0945-44f7-89f2-f6614dd406db-kube-api-access-hkwgh\") pod \"isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:22:53.878756 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.878742 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/551ab150-0945-44f7-89f2-f6614dd406db-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:22:53.879261 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.879241 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/551ab150-0945-44f7-89f2-f6614dd406db-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:22:53.879261 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.879255 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/551ab150-0945-44f7-89f2-f6614dd406db-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:22:53.881030 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.881001 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/551ab150-0945-44f7-89f2-f6614dd406db-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:22:53.895191 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:53.895167 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkwgh\" (UniqueName: \"kubernetes.io/projected/551ab150-0945-44f7-89f2-f6614dd406db-kube-api-access-hkwgh\") pod \"isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:22:54.061871 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:54.061784 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:22:54.342150 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:54.341932 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr"] Oct 31 01:22:54.344682 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:22:54.344640 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551ab150_0945_44f7_89f2_f6614dd406db.slice/crio-6351cef1b0937416a2a67ff52e1e0da2f232350e98cc756d3545cd4d18189598 WatchSource:0}: Error finding container 6351cef1b0937416a2a67ff52e1e0da2f232350e98cc756d3545cd4d18189598: Status 404 returned error can't find the container with id 6351cef1b0937416a2a67ff52e1e0da2f232350e98cc756d3545cd4d18189598 Oct 31 01:22:54.412959 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:54.412923 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" event={"ID":"551ab150-0945-44f7-89f2-f6614dd406db","Type":"ContainerStarted","Data":"a116eb49a5aa9d9c725ad7b4dae5a21b4ebd5221da5858aed919366705df5a95"} Oct 31 01:22:54.413130 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:54.412986 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" event={"ID":"551ab150-0945-44f7-89f2-f6614dd406db","Type":"ContainerStarted","Data":"6351cef1b0937416a2a67ff52e1e0da2f232350e98cc756d3545cd4d18189598"} Oct 31 01:22:58.430264 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:58.430222 5939 generic.go:356] "Generic (PLEG): container finished" podID="551ab150-0945-44f7-89f2-f6614dd406db" containerID="a116eb49a5aa9d9c725ad7b4dae5a21b4ebd5221da5858aed919366705df5a95" exitCode=0 Oct 31 01:22:58.430729 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:58.430291 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" event={"ID":"551ab150-0945-44f7-89f2-f6614dd406db","Type":"ContainerDied","Data":"a116eb49a5aa9d9c725ad7b4dae5a21b4ebd5221da5858aed919366705df5a95"} Oct 31 01:22:58.431562 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:22:58.431548 5939 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:23:00.515589 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:23:00.515548 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:23:05.464587 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:23:05.464552 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" event={"ID":"551ab150-0945-44f7-89f2-f6614dd406db","Type":"ContainerStarted","Data":"eac3e1092d09a270e3002d3b4d953d849fc33b01e3f623b7d1a753294ef5adc2"} Oct 31 01:23:05.465005 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:23:05.464611 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" event={"ID":"551ab150-0945-44f7-89f2-f6614dd406db","Type":"ContainerStarted","Data":"7dc4db99893e95a3d07a79745bf4db406ca2040b75a3b7a9364d8ced1b69b9d9"} Oct 31 01:23:05.465005 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:23:05.464809 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:23:05.492461 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:23:05.492405 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" podStartSLOduration=2.151401249 podStartE2EDuration="12.49238824s" totalImagesPullingTime="6.422121475s" totalInitContainerRuntime="3.918865516s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:05.492007286 +0000 UTC m=+3197.981282465" watchObservedRunningTime="2026-10-31 01:23:05.49238824 +0000 UTC m=+3197.981663420" Oct 31 01:23:06.468245 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:23:06.468210 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:23:06.469339 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:23:06.469310 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused" Oct 31 01:23:07.471894 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:23:07.471850 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused" Oct 31 01:23:12.476653 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:23:12.476624 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:23:12.477247 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:23:12.477218 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused" Oct 31 01:23:22.477893 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:23:22.477850 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused" Oct 31 01:23:30.509103 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:23:30.509066 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:23:32.477374 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:23:32.477338 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused" Oct 31 01:23:42.477335 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:23:42.477294 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused" Oct 31 01:23:52.477484 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:23:52.477438 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused" Oct 31 01:24:00.509863 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:24:00.509780 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:24:02.477307 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:02.477269 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused" Oct 31 01:24:12.478086 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:12.478040 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused" Oct 31 01:24:22.477884 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:22.477841 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused" Oct 31 01:24:30.509953 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:24:30.509907 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:24:32.477437 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:32.477397 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused" Oct 31 01:24:41.651559 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:41.651528 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:24:45.425224 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:45.425187 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr"] Oct 31 01:24:45.425639 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:45.425421 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kserve-container" containerID="cri-o://7dc4db99893e95a3d07a79745bf4db406ca2040b75a3b7a9364d8ced1b69b9d9" gracePeriod=30 Oct 31 01:24:45.425639 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:45.425505 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kube-rbac-proxy" containerID="cri-o://eac3e1092d09a270e3002d3b4d953d849fc33b01e3f623b7d1a753294ef5adc2" gracePeriod=30 Oct 31 01:24:45.871372 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:45.871336 5939 generic.go:356] "Generic (PLEG): container finished" podID="551ab150-0945-44f7-89f2-f6614dd406db" containerID="eac3e1092d09a270e3002d3b4d953d849fc33b01e3f623b7d1a753294ef5adc2" exitCode=0 Oct 31 01:24:45.871530 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:45.871409 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" event={"ID":"551ab150-0945-44f7-89f2-f6614dd406db","Type":"ContainerDied","Data":"eac3e1092d09a270e3002d3b4d953d849fc33b01e3f623b7d1a753294ef5adc2"} Oct 31 01:24:47.472098 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:47.472053 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused" Oct 31 01:24:48.030813 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:48.030779 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 01:24:48.033365 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:48.033343 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 01:24:50.893234 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:50.893149 5939 generic.go:356] "Generic (PLEG): container finished" podID="551ab150-0945-44f7-89f2-f6614dd406db" containerID="7dc4db99893e95a3d07a79745bf4db406ca2040b75a3b7a9364d8ced1b69b9d9" exitCode=0 Oct 31 01:24:50.893234 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:50.893223 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" event={"ID":"551ab150-0945-44f7-89f2-f6614dd406db","Type":"ContainerDied","Data":"7dc4db99893e95a3d07a79745bf4db406ca2040b75a3b7a9364d8ced1b69b9d9"} Oct 31 01:24:51.265434 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.265397 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:24:51.415496 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.415453 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551ab150-0945-44f7-89f2-f6614dd406db-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/551ab150-0945-44f7-89f2-f6614dd406db-kserve-provision-location\") pod \"551ab150-0945-44f7-89f2-f6614dd406db\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " Oct 31 01:24:51.415496 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.415504 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551ab150-0945-44f7-89f2-f6614dd406db-kube-api-access-hkwgh\" (UniqueName: \"kubernetes.io/projected/551ab150-0945-44f7-89f2-f6614dd406db-kube-api-access-hkwgh\") pod \"551ab150-0945-44f7-89f2-f6614dd406db\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " Oct 31 01:24:51.415747 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.415544 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/551ab150-0945-44f7-89f2-f6614dd406db-proxy-tls\" (UniqueName: \"kubernetes.io/secret/551ab150-0945-44f7-89f2-f6614dd406db-proxy-tls\") pod \"551ab150-0945-44f7-89f2-f6614dd406db\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " Oct 31 01:24:51.415747 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.415581 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551ab150-0945-44f7-89f2-f6614dd406db-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/551ab150-0945-44f7-89f2-f6614dd406db-cabundle-cert\") pod \"551ab150-0945-44f7-89f2-f6614dd406db\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " Oct 31 01:24:51.415747 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.415641 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551ab150-0945-44f7-89f2-f6614dd406db-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/551ab150-0945-44f7-89f2-f6614dd406db-kube-rbac-proxy-sar-config\") pod \"551ab150-0945-44f7-89f2-f6614dd406db\" (UID: \"551ab150-0945-44f7-89f2-f6614dd406db\") " Oct 31 01:24:51.415928 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.415873 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551ab150-0945-44f7-89f2-f6614dd406db-kserve-provision-location" pod "551ab150-0945-44f7-89f2-f6614dd406db" (UID: "551ab150-0945-44f7-89f2-f6614dd406db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:51.416052 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.416031 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551ab150-0945-44f7-89f2-f6614dd406db-cabundle-cert" pod "551ab150-0945-44f7-89f2-f6614dd406db" (UID: "551ab150-0945-44f7-89f2-f6614dd406db"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:24:51.416137 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.416113 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551ab150-0945-44f7-89f2-f6614dd406db-kube-rbac-proxy-sar-config" pod "551ab150-0945-44f7-89f2-f6614dd406db" (UID: "551ab150-0945-44f7-89f2-f6614dd406db"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:24:51.417792 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.417771 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/551ab150-0945-44f7-89f2-f6614dd406db-proxy-tls" pod "551ab150-0945-44f7-89f2-f6614dd406db" (UID: "551ab150-0945-44f7-89f2-f6614dd406db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:24:51.417861 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.417841 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551ab150-0945-44f7-89f2-f6614dd406db-kube-api-access-hkwgh" pod "551ab150-0945-44f7-89f2-f6614dd406db" (UID: "551ab150-0945-44f7-89f2-f6614dd406db"). InnerVolumeSpecName "kube-api-access-hkwgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:51.516602 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.516495 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/551ab150-0945-44f7-89f2-f6614dd406db-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:24:51.516602 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.516530 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/551ab150-0945-44f7-89f2-f6614dd406db-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:24:51.516602 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.516550 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/551ab150-0945-44f7-89f2-f6614dd406db-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:24:51.516602 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.516562 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/551ab150-0945-44f7-89f2-f6614dd406db-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:24:51.516602 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.516574 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkwgh\" (UniqueName: \"kubernetes.io/projected/551ab150-0945-44f7-89f2-f6614dd406db-kube-api-access-hkwgh\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:24:51.899820 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.899784 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" event={"ID":"551ab150-0945-44f7-89f2-f6614dd406db","Type":"ContainerDied","Data":"6351cef1b0937416a2a67ff52e1e0da2f232350e98cc756d3545cd4d18189598"} Oct 31 01:24:51.900379 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.899831 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr" Oct 31 01:24:51.900379 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.899855 5939 scope.go:122] "RemoveContainer" containerID="eac3e1092d09a270e3002d3b4d953d849fc33b01e3f623b7d1a753294ef5adc2" Oct 31 01:24:51.908340 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.908314 5939 scope.go:122] "RemoveContainer" containerID="7dc4db99893e95a3d07a79745bf4db406ca2040b75a3b7a9364d8ced1b69b9d9" Oct 31 01:24:51.916117 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.916095 5939 scope.go:122] "RemoveContainer" containerID="a116eb49a5aa9d9c725ad7b4dae5a21b4ebd5221da5858aed919366705df5a95" Oct 31 01:24:51.925945 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.925924 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr"] Oct 31 01:24:51.931532 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:51.931511 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-runtime-predictor-668848dbb9-gt4rr"] Oct 31 01:24:52.885479 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.885447 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47"] Oct 31 01:24:52.885792 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.885779 5939 cpu_manager.go:414] "containerMap: removing container" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kube-rbac-proxy" Oct 31 01:24:52.885852 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.885793 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kube-rbac-proxy" Oct 31 01:24:52.885852 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.885806 5939 cpu_manager.go:414] "containerMap: removing container" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kserve-container" Oct 31 01:24:52.885852 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.885811 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kserve-container" Oct 31 01:24:52.885852 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.885820 5939 cpu_manager.go:414] "containerMap: removing container" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="storage-initializer" Oct 31 01:24:52.885852 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.885825 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="storage-initializer" Oct 31 01:24:52.886021 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.885918 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kube-rbac-proxy" Oct 31 01:24:52.886021 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.885927 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="551ab150-0945-44f7-89f2-f6614dd406db" containerName="kserve-container" Oct 31 01:24:52.890485 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.890465 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:52.898646 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.898627 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\"" Oct 31 01:24:52.900441 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.900423 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\"" Oct 31 01:24:52.901915 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.901893 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 01:24:52.901915 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.901909 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\"" Oct 31 01:24:52.902135 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.901902 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\"" Oct 31 01:24:52.902135 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.901937 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\"" Oct 31 01:24:52.905138 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.905118 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47"] Oct 31 01:24:52.998642 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:52.998616 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\"" Oct 31 01:24:53.031025 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.030999 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:53.031144 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.031033 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6pv2\" (UniqueName: \"kubernetes.io/projected/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kube-api-access-l6pv2\") pod \"isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:53.031144 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.031058 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d16d2e09-8b96-4f90-87d3-3d2b5a445551-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:53.031286 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.031264 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:53.031373 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.031360 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d16d2e09-8b96-4f90-87d3-3d2b5a445551-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:53.132010 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.131974 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d16d2e09-8b96-4f90-87d3-3d2b5a445551-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:53.132188 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.132017 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:53.132188 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.132035 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6pv2\" (UniqueName: \"kubernetes.io/projected/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kube-api-access-l6pv2\") pod \"isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:53.132188 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.132058 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d16d2e09-8b96-4f90-87d3-3d2b5a445551-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:53.132188 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.132144 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:53.132636 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.132610 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:53.132810 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.132790 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d16d2e09-8b96-4f90-87d3-3d2b5a445551-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:53.132877 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.132817 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:53.134480 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.134461 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d16d2e09-8b96-4f90-87d3-3d2b5a445551-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:53.148484 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.148431 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6pv2\" (UniqueName: \"kubernetes.io/projected/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kube-api-access-l6pv2\") pod \"isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:53.209990 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.209969 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:53.495251 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.495180 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47"] Oct 31 01:24:53.498490 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:24:53.498449 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16d2e09_8b96_4f90_87d3_3d2b5a445551.slice/crio-165fa1ad29c5544345a4a4608695fca833ad383c73a4fa83d59c01affe6e0fdb WatchSource:0}: Error finding container 165fa1ad29c5544345a4a4608695fca833ad383c73a4fa83d59c01affe6e0fdb: Status 404 returned error can't find the container with id 165fa1ad29c5544345a4a4608695fca833ad383c73a4fa83d59c01affe6e0fdb Oct 31 01:24:53.653457 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.653420 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551ab150-0945-44f7-89f2-f6614dd406db" path="/var/lib/kubelet/pods/551ab150-0945-44f7-89f2-f6614dd406db/volumes" Oct 31 01:24:53.911565 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.911526 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" event={"ID":"d16d2e09-8b96-4f90-87d3-3d2b5a445551","Type":"ContainerStarted","Data":"23eb071cb68c0ece4b2d964e6c7e4dfc0cde96f3826c3377d20e42896bd360b0"} Oct 31 01:24:53.911972 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:53.911588 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" event={"ID":"d16d2e09-8b96-4f90-87d3-3d2b5a445551","Type":"ContainerStarted","Data":"165fa1ad29c5544345a4a4608695fca833ad383c73a4fa83d59c01affe6e0fdb"} Oct 31 01:24:57.930045 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:57.929943 5939 generic.go:356] "Generic (PLEG): container finished" podID="d16d2e09-8b96-4f90-87d3-3d2b5a445551" containerID="23eb071cb68c0ece4b2d964e6c7e4dfc0cde96f3826c3377d20e42896bd360b0" exitCode=0 Oct 31 01:24:57.930045 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:57.930019 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" event={"ID":"d16d2e09-8b96-4f90-87d3-3d2b5a445551","Type":"ContainerDied","Data":"23eb071cb68c0ece4b2d964e6c7e4dfc0cde96f3826c3377d20e42896bd360b0"} Oct 31 01:24:58.937058 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:58.937025 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" event={"ID":"d16d2e09-8b96-4f90-87d3-3d2b5a445551","Type":"ContainerStarted","Data":"c38f822ad0eeebca351442af6b3badfc5b782620ff3c12f9a0d92c12e2f492f2"} Oct 31 01:24:58.937467 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:58.937084 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" event={"ID":"d16d2e09-8b96-4f90-87d3-3d2b5a445551","Type":"ContainerStarted","Data":"924c74cdf559a018b34d0562e5f98c1dadf1c70c03cfb479f91b8743c9810f82"} Oct 31 01:24:58.937467 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:58.937278 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:58.937467 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:58.937333 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:24:58.965822 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:24:58.965773 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" podStartSLOduration=2.8551770530000002 podStartE2EDuration="6.965756878s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.110579825s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:58.96406627 +0000 UTC m=+3311.453341448" watchObservedRunningTime="2026-10-31 01:24:58.965756878 +0000 UTC m=+3311.455032057" Oct 31 01:25:00.509354 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:25:00.509313 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:25:04.947423 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:04.947394 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:25:30.509182 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:25:30.509095 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:25:34.951410 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:34.951379 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:25:43.188895 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:43.188857 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47"] Oct 31 01:25:43.189381 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:43.189227 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" podUID="d16d2e09-8b96-4f90-87d3-3d2b5a445551" containerName="kserve-container" containerID="cri-o://924c74cdf559a018b34d0562e5f98c1dadf1c70c03cfb479f91b8743c9810f82" gracePeriod=30 Oct 31 01:25:43.189381 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:43.189259 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" podUID="d16d2e09-8b96-4f90-87d3-3d2b5a445551" containerName="kube-rbac-proxy" containerID="cri-o://c38f822ad0eeebca351442af6b3badfc5b782620ff3c12f9a0d92c12e2f492f2" gracePeriod=30 Oct 31 01:25:44.134772 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.134725 5939 generic.go:356] "Generic (PLEG): container finished" podID="d16d2e09-8b96-4f90-87d3-3d2b5a445551" containerID="c38f822ad0eeebca351442af6b3badfc5b782620ff3c12f9a0d92c12e2f492f2" exitCode=0 Oct 31 01:25:44.134772 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.134769 5939 generic.go:356] "Generic (PLEG): container finished" podID="d16d2e09-8b96-4f90-87d3-3d2b5a445551" containerID="924c74cdf559a018b34d0562e5f98c1dadf1c70c03cfb479f91b8743c9810f82" exitCode=0 Oct 31 01:25:44.134973 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.134809 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" event={"ID":"d16d2e09-8b96-4f90-87d3-3d2b5a445551","Type":"ContainerDied","Data":"c38f822ad0eeebca351442af6b3badfc5b782620ff3c12f9a0d92c12e2f492f2"} Oct 31 01:25:44.134973 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.134889 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" event={"ID":"d16d2e09-8b96-4f90-87d3-3d2b5a445551","Type":"ContainerDied","Data":"924c74cdf559a018b34d0562e5f98c1dadf1c70c03cfb479f91b8743c9810f82"} Oct 31 01:25:44.234004 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.233978 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:25:44.300764 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.300688 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kube-rbac-proxy-sar-config\") pod \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " Oct 31 01:25:44.300764 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.300752 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d16d2e09-8b96-4f90-87d3-3d2b5a445551-proxy-tls\" (UniqueName: \"kubernetes.io/secret/d16d2e09-8b96-4f90-87d3-3d2b5a445551-proxy-tls\") pod \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " Oct 31 01:25:44.300936 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.300832 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16d2e09-8b96-4f90-87d3-3d2b5a445551-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d16d2e09-8b96-4f90-87d3-3d2b5a445551-cabundle-cert\") pod \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " Oct 31 01:25:44.300936 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.300876 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kserve-provision-location\") pod \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " Oct 31 01:25:44.300936 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.300921 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kube-api-access-l6pv2\" (UniqueName: \"kubernetes.io/projected/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kube-api-access-l6pv2\") pod \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\" (UID: \"d16d2e09-8b96-4f90-87d3-3d2b5a445551\") " Oct 31 01:25:44.301192 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.301170 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kube-rbac-proxy-sar-config" pod "d16d2e09-8b96-4f90-87d3-3d2b5a445551" (UID: "d16d2e09-8b96-4f90-87d3-3d2b5a445551"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:25:44.301274 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.301232 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16d2e09-8b96-4f90-87d3-3d2b5a445551-cabundle-cert" pod "d16d2e09-8b96-4f90-87d3-3d2b5a445551" (UID: "d16d2e09-8b96-4f90-87d3-3d2b5a445551"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:25:44.301274 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.301232 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kserve-provision-location" pod "d16d2e09-8b96-4f90-87d3-3d2b5a445551" (UID: "d16d2e09-8b96-4f90-87d3-3d2b5a445551"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:25:44.302952 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.302925 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16d2e09-8b96-4f90-87d3-3d2b5a445551-proxy-tls" pod "d16d2e09-8b96-4f90-87d3-3d2b5a445551" (UID: "d16d2e09-8b96-4f90-87d3-3d2b5a445551"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:25:44.303066 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.303016 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kube-api-access-l6pv2" pod "d16d2e09-8b96-4f90-87d3-3d2b5a445551" (UID: "d16d2e09-8b96-4f90-87d3-3d2b5a445551"). InnerVolumeSpecName "kube-api-access-l6pv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:44.401523 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.401488 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d16d2e09-8b96-4f90-87d3-3d2b5a445551-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:44.401523 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.401517 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:44.401523 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.401530 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6pv2\" (UniqueName: \"kubernetes.io/projected/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kube-api-access-l6pv2\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:44.401768 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.401540 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d16d2e09-8b96-4f90-87d3-3d2b5a445551-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:44.401768 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:44.401553 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d16d2e09-8b96-4f90-87d3-3d2b5a445551-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:45.140636 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:45.140612 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" Oct 31 01:25:45.140817 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:45.140605 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47" event={"ID":"d16d2e09-8b96-4f90-87d3-3d2b5a445551","Type":"ContainerDied","Data":"165fa1ad29c5544345a4a4608695fca833ad383c73a4fa83d59c01affe6e0fdb"} Oct 31 01:25:45.140817 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:45.140741 5939 scope.go:122] "RemoveContainer" containerID="c38f822ad0eeebca351442af6b3badfc5b782620ff3c12f9a0d92c12e2f492f2" Oct 31 01:25:45.149116 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:45.149095 5939 scope.go:122] "RemoveContainer" containerID="924c74cdf559a018b34d0562e5f98c1dadf1c70c03cfb479f91b8743c9810f82" Oct 31 01:25:45.156596 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:45.156582 5939 scope.go:122] "RemoveContainer" containerID="23eb071cb68c0ece4b2d964e6c7e4dfc0cde96f3826c3377d20e42896bd360b0" Oct 31 01:25:45.166639 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:45.166620 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47"] Oct 31 01:25:45.172825 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:45.172799 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-lightgbm-v2-runtime-predictor-849db5d649-mgv47"] Oct 31 01:25:45.652391 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:25:45.652361 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d16d2e09-8b96-4f90-87d3-3d2b5a445551" path="/var/lib/kubelet/pods/d16d2e09-8b96-4f90-87d3-3d2b5a445551/volumes" Oct 31 01:26:00.508982 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:26:00.508939 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:26:20.427607 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.427577 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz"] Oct 31 01:26:20.428093 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.427919 5939 cpu_manager.go:414] "containerMap: removing container" podUID="d16d2e09-8b96-4f90-87d3-3d2b5a445551" containerName="storage-initializer" Oct 31 01:26:20.428093 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.427932 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16d2e09-8b96-4f90-87d3-3d2b5a445551" containerName="storage-initializer" Oct 31 01:26:20.428093 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.427952 5939 cpu_manager.go:414] "containerMap: removing container" podUID="d16d2e09-8b96-4f90-87d3-3d2b5a445551" containerName="kube-rbac-proxy" Oct 31 01:26:20.428093 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.427959 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16d2e09-8b96-4f90-87d3-3d2b5a445551" containerName="kube-rbac-proxy" Oct 31 01:26:20.428093 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.427976 5939 cpu_manager.go:414] "containerMap: removing container" podUID="d16d2e09-8b96-4f90-87d3-3d2b5a445551" containerName="kserve-container" Oct 31 01:26:20.428093 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.427981 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16d2e09-8b96-4f90-87d3-3d2b5a445551" containerName="kserve-container" Oct 31 01:26:20.428093 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.428048 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16d2e09-8b96-4f90-87d3-3d2b5a445551" containerName="kube-rbac-proxy" Oct 31 01:26:20.428093 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.428059 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16d2e09-8b96-4f90-87d3-3d2b5a445551" containerName="kserve-container" Oct 31 01:26:20.431374 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.431358 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:20.440872 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.440851 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\"" Oct 31 01:26:20.440979 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.440909 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\"" Oct 31 01:26:20.440979 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.440917 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 01:26:20.441885 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.441862 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\"" Oct 31 01:26:20.441942 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.441929 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\"" Oct 31 01:26:20.441981 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.441935 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\"" Oct 31 01:26:20.441981 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.441961 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\"" Oct 31 01:26:20.446836 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.446807 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz"] Oct 31 01:26:20.511318 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.511291 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40b4abbb-e732-4c66-94db-8380143766b8-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:20.511441 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.511321 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40b4abbb-e732-4c66-94db-8380143766b8-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:20.511441 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.511347 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40b4abbb-e732-4c66-94db-8380143766b8-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:20.511441 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.511423 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfvkd\" (UniqueName: \"kubernetes.io/projected/40b4abbb-e732-4c66-94db-8380143766b8-kube-api-access-jfvkd\") pod \"isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:20.511618 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.511496 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40b4abbb-e732-4c66-94db-8380143766b8-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:20.612500 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.612475 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40b4abbb-e732-4c66-94db-8380143766b8-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:20.612604 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.612502 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40b4abbb-e732-4c66-94db-8380143766b8-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:20.612604 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.612522 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40b4abbb-e732-4c66-94db-8380143766b8-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:20.612604 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.612545 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvkd\" (UniqueName: \"kubernetes.io/projected/40b4abbb-e732-4c66-94db-8380143766b8-kube-api-access-jfvkd\") pod \"isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:20.612604 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.612568 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40b4abbb-e732-4c66-94db-8380143766b8-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:20.612981 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.612963 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40b4abbb-e732-4c66-94db-8380143766b8-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:20.613276 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.613253 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40b4abbb-e732-4c66-94db-8380143766b8-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:20.613320 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.613253 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40b4abbb-e732-4c66-94db-8380143766b8-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:20.614901 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.614881 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40b4abbb-e732-4c66-94db-8380143766b8-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:20.626863 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.626845 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfvkd\" (UniqueName: \"kubernetes.io/projected/40b4abbb-e732-4c66-94db-8380143766b8-kube-api-access-jfvkd\") pod \"isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:20.749923 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:20.749844 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:21.034853 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:21.034771 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz"] Oct 31 01:26:21.038128 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:26:21.038082 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b4abbb_e732_4c66_94db_8380143766b8.slice/crio-23de734e4381b35d6ee72ccfc92623d4197889ccbd2e94d1cd4ca8a78261b6ff WatchSource:0}: Error finding container 23de734e4381b35d6ee72ccfc92623d4197889ccbd2e94d1cd4ca8a78261b6ff: Status 404 returned error can't find the container with id 23de734e4381b35d6ee72ccfc92623d4197889ccbd2e94d1cd4ca8a78261b6ff Oct 31 01:26:21.289426 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:21.289336 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" event={"ID":"40b4abbb-e732-4c66-94db-8380143766b8","Type":"ContainerStarted","Data":"b01a08e8cb722cbd5168b08914b9614354518f18cc37b057d8cb6d54e3ce60a6"} Oct 31 01:26:21.289426 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:21.289418 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" event={"ID":"40b4abbb-e732-4c66-94db-8380143766b8","Type":"ContainerStarted","Data":"23de734e4381b35d6ee72ccfc92623d4197889ccbd2e94d1cd4ca8a78261b6ff"} Oct 31 01:26:26.310037 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:26.310005 5939 generic.go:356] "Generic (PLEG): container finished" podID="40b4abbb-e732-4c66-94db-8380143766b8" containerID="b01a08e8cb722cbd5168b08914b9614354518f18cc37b057d8cb6d54e3ce60a6" exitCode=0 Oct 31 01:26:26.310436 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:26.310091 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" event={"ID":"40b4abbb-e732-4c66-94db-8380143766b8","Type":"ContainerDied","Data":"b01a08e8cb722cbd5168b08914b9614354518f18cc37b057d8cb6d54e3ce60a6"} Oct 31 01:26:27.316637 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:27.316592 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" event={"ID":"40b4abbb-e732-4c66-94db-8380143766b8","Type":"ContainerStarted","Data":"631f471f84293b5fa4876e910c2d7d1e111f451e132d534f7175b9dff5ab4233"} Oct 31 01:26:27.317174 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:27.316698 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" event={"ID":"40b4abbb-e732-4c66-94db-8380143766b8","Type":"ContainerStarted","Data":"40716a4a45120f8853a84e3a939e6e99dcbb775368dd137e717de803358007ea"} Oct 31 01:26:27.317174 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:27.316897 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:27.346347 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:27.346295 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" podStartSLOduration=3.11444192 podStartE2EDuration="7.346283344s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.231841424s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:27.344777985 +0000 UTC m=+3399.834053165" watchObservedRunningTime="2026-10-31 01:26:27.346283344 +0000 UTC m=+3399.835558522" Oct 31 01:26:28.322122 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:28.322091 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:26:30.508693 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:26:30.508641 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:26:34.331029 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:26:34.331001 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:27:00.508511 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:27:00.508430 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:27:04.335583 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:04.335554 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:27:10.729957 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:10.729928 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz"] Oct 31 01:27:10.730328 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:10.730157 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" podUID="40b4abbb-e732-4c66-94db-8380143766b8" containerName="kserve-container" containerID="cri-o://40716a4a45120f8853a84e3a939e6e99dcbb775368dd137e717de803358007ea" gracePeriod=30 Oct 31 01:27:10.730328 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:10.730186 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" podUID="40b4abbb-e732-4c66-94db-8380143766b8" containerName="kube-rbac-proxy" containerID="cri-o://631f471f84293b5fa4876e910c2d7d1e111f451e132d534f7175b9dff5ab4233" gracePeriod=30 Oct 31 01:27:11.493126 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:11.493093 5939 generic.go:356] "Generic (PLEG): container finished" podID="40b4abbb-e732-4c66-94db-8380143766b8" containerID="631f471f84293b5fa4876e910c2d7d1e111f451e132d534f7175b9dff5ab4233" exitCode=0 Oct 31 01:27:11.493574 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:11.493165 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" event={"ID":"40b4abbb-e732-4c66-94db-8380143766b8","Type":"ContainerDied","Data":"631f471f84293b5fa4876e910c2d7d1e111f451e132d534f7175b9dff5ab4233"} Oct 31 01:27:11.867855 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:11.867832 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:27:11.947741 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:11.947715 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b4abbb-e732-4c66-94db-8380143766b8-kube-api-access-jfvkd\" (UniqueName: \"kubernetes.io/projected/40b4abbb-e732-4c66-94db-8380143766b8-kube-api-access-jfvkd\") pod \"40b4abbb-e732-4c66-94db-8380143766b8\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " Oct 31 01:27:11.947870 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:11.947770 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b4abbb-e732-4c66-94db-8380143766b8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40b4abbb-e732-4c66-94db-8380143766b8-kube-rbac-proxy-sar-config\") pod \"40b4abbb-e732-4c66-94db-8380143766b8\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " Oct 31 01:27:11.947870 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:11.947790 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b4abbb-e732-4c66-94db-8380143766b8-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40b4abbb-e732-4c66-94db-8380143766b8-cabundle-cert\") pod \"40b4abbb-e732-4c66-94db-8380143766b8\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " Oct 31 01:27:11.947870 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:11.947837 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b4abbb-e732-4c66-94db-8380143766b8-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40b4abbb-e732-4c66-94db-8380143766b8-kserve-provision-location\") pod \"40b4abbb-e732-4c66-94db-8380143766b8\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " Oct 31 01:27:11.947870 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:11.947864 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b4abbb-e732-4c66-94db-8380143766b8-proxy-tls\" (UniqueName: \"kubernetes.io/secret/40b4abbb-e732-4c66-94db-8380143766b8-proxy-tls\") pod \"40b4abbb-e732-4c66-94db-8380143766b8\" (UID: \"40b4abbb-e732-4c66-94db-8380143766b8\") " Oct 31 01:27:11.948192 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:11.948170 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b4abbb-e732-4c66-94db-8380143766b8-cabundle-cert" pod "40b4abbb-e732-4c66-94db-8380143766b8" (UID: "40b4abbb-e732-4c66-94db-8380143766b8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:11.948292 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:11.948183 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b4abbb-e732-4c66-94db-8380143766b8-kube-rbac-proxy-sar-config" pod "40b4abbb-e732-4c66-94db-8380143766b8" (UID: "40b4abbb-e732-4c66-94db-8380143766b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:11.948292 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:11.948195 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b4abbb-e732-4c66-94db-8380143766b8-kserve-provision-location" pod "40b4abbb-e732-4c66-94db-8380143766b8" (UID: "40b4abbb-e732-4c66-94db-8380143766b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:11.949975 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:11.949948 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b4abbb-e732-4c66-94db-8380143766b8-kube-api-access-jfvkd" pod "40b4abbb-e732-4c66-94db-8380143766b8" (UID: "40b4abbb-e732-4c66-94db-8380143766b8"). InnerVolumeSpecName "kube-api-access-jfvkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:11.950071 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:11.950027 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b4abbb-e732-4c66-94db-8380143766b8-proxy-tls" pod "40b4abbb-e732-4c66-94db-8380143766b8" (UID: "40b4abbb-e732-4c66-94db-8380143766b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:12.049642 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.049556 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfvkd\" (UniqueName: \"kubernetes.io/projected/40b4abbb-e732-4c66-94db-8380143766b8-kube-api-access-jfvkd\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:12.049642 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.049581 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40b4abbb-e732-4c66-94db-8380143766b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:12.049642 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.049605 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40b4abbb-e732-4c66-94db-8380143766b8-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:12.049642 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.049614 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40b4abbb-e732-4c66-94db-8380143766b8-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:12.049642 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.049624 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40b4abbb-e732-4c66-94db-8380143766b8-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:12.504446 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.504412 5939 generic.go:356] "Generic (PLEG): container finished" podID="40b4abbb-e732-4c66-94db-8380143766b8" containerID="40716a4a45120f8853a84e3a939e6e99dcbb775368dd137e717de803358007ea" exitCode=0 Oct 31 01:27:12.504634 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.504477 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" Oct 31 01:27:12.504634 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.504477 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" event={"ID":"40b4abbb-e732-4c66-94db-8380143766b8","Type":"ContainerDied","Data":"40716a4a45120f8853a84e3a939e6e99dcbb775368dd137e717de803358007ea"} Oct 31 01:27:12.504792 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.504639 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz" event={"ID":"40b4abbb-e732-4c66-94db-8380143766b8","Type":"ContainerDied","Data":"23de734e4381b35d6ee72ccfc92623d4197889ccbd2e94d1cd4ca8a78261b6ff"} Oct 31 01:27:12.504792 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.504700 5939 scope.go:122] "RemoveContainer" containerID="631f471f84293b5fa4876e910c2d7d1e111f451e132d534f7175b9dff5ab4233" Oct 31 01:27:12.514457 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.514443 5939 scope.go:122] "RemoveContainer" containerID="40716a4a45120f8853a84e3a939e6e99dcbb775368dd137e717de803358007ea" Oct 31 01:27:12.522122 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.522105 5939 scope.go:122] "RemoveContainer" containerID="b01a08e8cb722cbd5168b08914b9614354518f18cc37b057d8cb6d54e3ce60a6" Oct 31 01:27:12.528918 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.528901 5939 scope.go:122] "RemoveContainer" containerID="631f471f84293b5fa4876e910c2d7d1e111f451e132d534f7175b9dff5ab4233" Oct 31 01:27:12.529161 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:27:12.529141 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"631f471f84293b5fa4876e910c2d7d1e111f451e132d534f7175b9dff5ab4233\": container with ID starting with 631f471f84293b5fa4876e910c2d7d1e111f451e132d534f7175b9dff5ab4233 not found: ID does not exist" containerID="631f471f84293b5fa4876e910c2d7d1e111f451e132d534f7175b9dff5ab4233" Oct 31 01:27:12.529210 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.529170 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631f471f84293b5fa4876e910c2d7d1e111f451e132d534f7175b9dff5ab4233"} err="failed to get container status \"631f471f84293b5fa4876e910c2d7d1e111f451e132d534f7175b9dff5ab4233\": rpc error: code = NotFound desc = could not find container \"631f471f84293b5fa4876e910c2d7d1e111f451e132d534f7175b9dff5ab4233\": container with ID starting with 631f471f84293b5fa4876e910c2d7d1e111f451e132d534f7175b9dff5ab4233 not found: ID does not exist" Oct 31 01:27:12.529210 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.529186 5939 scope.go:122] "RemoveContainer" containerID="40716a4a45120f8853a84e3a939e6e99dcbb775368dd137e717de803358007ea" Oct 31 01:27:12.529410 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:27:12.529388 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40716a4a45120f8853a84e3a939e6e99dcbb775368dd137e717de803358007ea\": container with ID starting with 40716a4a45120f8853a84e3a939e6e99dcbb775368dd137e717de803358007ea not found: ID does not exist" containerID="40716a4a45120f8853a84e3a939e6e99dcbb775368dd137e717de803358007ea" Oct 31 01:27:12.529481 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.529420 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40716a4a45120f8853a84e3a939e6e99dcbb775368dd137e717de803358007ea"} err="failed to get container status \"40716a4a45120f8853a84e3a939e6e99dcbb775368dd137e717de803358007ea\": rpc error: code = NotFound desc = could not find container \"40716a4a45120f8853a84e3a939e6e99dcbb775368dd137e717de803358007ea\": container with ID starting with 40716a4a45120f8853a84e3a939e6e99dcbb775368dd137e717de803358007ea not found: ID does not exist" Oct 31 01:27:12.529481 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.529441 5939 scope.go:122] "RemoveContainer" containerID="b01a08e8cb722cbd5168b08914b9614354518f18cc37b057d8cb6d54e3ce60a6" Oct 31 01:27:12.529686 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:27:12.529655 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b01a08e8cb722cbd5168b08914b9614354518f18cc37b057d8cb6d54e3ce60a6\": container with ID starting with b01a08e8cb722cbd5168b08914b9614354518f18cc37b057d8cb6d54e3ce60a6 not found: ID does not exist" containerID="b01a08e8cb722cbd5168b08914b9614354518f18cc37b057d8cb6d54e3ce60a6" Oct 31 01:27:12.529730 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.529684 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01a08e8cb722cbd5168b08914b9614354518f18cc37b057d8cb6d54e3ce60a6"} err="failed to get container status \"b01a08e8cb722cbd5168b08914b9614354518f18cc37b057d8cb6d54e3ce60a6\": rpc error: code = NotFound desc = could not find container \"b01a08e8cb722cbd5168b08914b9614354518f18cc37b057d8cb6d54e3ce60a6\": container with ID starting with b01a08e8cb722cbd5168b08914b9614354518f18cc37b057d8cb6d54e3ce60a6 not found: ID does not exist" Oct 31 01:27:12.534732 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.534712 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz"] Oct 31 01:27:12.538963 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:12.538945 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-mlflow-v2-runtime-predictor-7ff575c5fd-fdbhz"] Oct 31 01:27:13.651868 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:13.651839 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b4abbb-e732-4c66-94db-8380143766b8" path="/var/lib/kubelet/pods/40b4abbb-e732-4c66-94db-8380143766b8/volumes" Oct 31 01:27:14.165382 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.165350 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr"] Oct 31 01:27:14.165795 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.165778 5939 cpu_manager.go:414] "containerMap: removing container" podUID="40b4abbb-e732-4c66-94db-8380143766b8" containerName="storage-initializer" Oct 31 01:27:14.165898 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.165796 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b4abbb-e732-4c66-94db-8380143766b8" containerName="storage-initializer" Oct 31 01:27:14.165898 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.165815 5939 cpu_manager.go:414] "containerMap: removing container" podUID="40b4abbb-e732-4c66-94db-8380143766b8" containerName="kube-rbac-proxy" Oct 31 01:27:14.165898 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.165822 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b4abbb-e732-4c66-94db-8380143766b8" containerName="kube-rbac-proxy" Oct 31 01:27:14.165898 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.165839 5939 cpu_manager.go:414] "containerMap: removing container" podUID="40b4abbb-e732-4c66-94db-8380143766b8" containerName="kserve-container" Oct 31 01:27:14.165898 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.165846 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b4abbb-e732-4c66-94db-8380143766b8" containerName="kserve-container" Oct 31 01:27:14.166147 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.165939 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b4abbb-e732-4c66-94db-8380143766b8" containerName="kube-rbac-proxy" Oct 31 01:27:14.166147 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.165952 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b4abbb-e732-4c66-94db-8380143766b8" containerName="kserve-container" Oct 31 01:27:14.170967 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.170944 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:14.176789 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.176765 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\"" Oct 31 01:27:14.176957 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.176914 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"kube-root-ca.crt\"" Oct 31 01:27:14.177035 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.176995 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"openshift-service-ca.crt\"" Oct 31 01:27:14.177035 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.177001 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 01:27:14.177919 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.177900 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"default-dockercfg-95zbn\"" Oct 31 01:27:14.177919 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.177916 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"seaweedfs-s3-creds\"" Oct 31 01:27:14.178051 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.177983 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-f4cda2dafd\"/\"isvc-sklearn-mcp-predictor-serving-cert\"" Oct 31 01:27:14.189765 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.189746 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr"] Oct 31 01:27:14.267956 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.267933 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:14.268086 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.267969 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:14.268086 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.268058 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:14.268172 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.268114 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7ppj\" (UniqueName: \"kubernetes.io/projected/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kube-api-access-b7ppj\") pod \"isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:14.268213 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.268180 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:14.369038 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.369012 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:14.369163 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.369045 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:14.369163 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.369065 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:14.369276 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.369203 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7ppj\" (UniqueName: \"kubernetes.io/projected/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kube-api-access-b7ppj\") pod \"isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:14.369276 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.369244 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:14.369422 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.369403 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:14.369756 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.369736 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:14.369890 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.369874 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:14.371513 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.371496 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:14.383362 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.383334 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7ppj\" (UniqueName: \"kubernetes.io/projected/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kube-api-access-b7ppj\") pod \"isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:14.490200 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.490103 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:14.776285 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:14.776206 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr"] Oct 31 01:27:14.779463 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:27:14.779434 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a635d7_7f4b_4dbb_b9cc_bc4df81ff47b.slice/crio-4b11dbbedb64e5523b4e0b29ade46317b1ebe735ae2335a4c6b9bba1eb42d23f WatchSource:0}: Error finding container 4b11dbbedb64e5523b4e0b29ade46317b1ebe735ae2335a4c6b9bba1eb42d23f: Status 404 returned error can't find the container with id 4b11dbbedb64e5523b4e0b29ade46317b1ebe735ae2335a4c6b9bba1eb42d23f Oct 31 01:27:15.519957 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:15.519923 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" event={"ID":"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b","Type":"ContainerStarted","Data":"fcecd32799d1fadf0f3d9cbf3eb601bb91f5a73b8b88da3312df4d10547e8ca3"} Oct 31 01:27:15.520132 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:15.519982 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" event={"ID":"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b","Type":"ContainerStarted","Data":"4b11dbbedb64e5523b4e0b29ade46317b1ebe735ae2335a4c6b9bba1eb42d23f"} Oct 31 01:27:19.538508 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:19.538467 5939 generic.go:356] "Generic (PLEG): container finished" podID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerID="fcecd32799d1fadf0f3d9cbf3eb601bb91f5a73b8b88da3312df4d10547e8ca3" exitCode=0 Oct 31 01:27:19.538914 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:19.538527 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" event={"ID":"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b","Type":"ContainerDied","Data":"fcecd32799d1fadf0f3d9cbf3eb601bb91f5a73b8b88da3312df4d10547e8ca3"} Oct 31 01:27:20.545522 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:20.545488 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" event={"ID":"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b","Type":"ContainerStarted","Data":"9087f570fc99872d4482731aaee56289b06c55a646a9500a2a3ad55cf1ca3ae3"} Oct 31 01:27:22.556200 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:22.556173 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" event={"ID":"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b","Type":"ContainerStarted","Data":"a5afe37433e2edf419a6d720bf954f8c94d83211d31f7f339a94315b9f7f1a54"} Oct 31 01:27:23.569698 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:23.569644 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" event={"ID":"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b","Type":"ContainerStarted","Data":"eb8057b3b2e44b52ef04694acfa545951441096d7980bd6a1f7b0382849badc0"} Oct 31 01:27:23.570115 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:23.569902 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:23.570115 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:23.569951 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:23.570115 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:23.569984 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:23.601783 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:23.601732 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" podStartSLOduration=2.868825098 podStartE2EDuration="9.601717293s" totalImagesPullingTime="2.856042335s" totalInitContainerRuntime="3.87684986s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:23.599192111 +0000 UTC m=+3456.088467312" watchObservedRunningTime="2026-10-31 01:27:23.601717293 +0000 UTC m=+3456.090992541" Oct 31 01:27:29.580909 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:29.580883 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:27:30.509429 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:27:30.509390 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:27:49.581938 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:49.581898 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.65:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.65:8080: connect: connection refused" Oct 31 01:27:54.491466 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:54.491420 5939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.65:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.65:8080: connect: connection refused" Oct 31 01:27:59.723090 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:27:59.723055 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:28:00.508924 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:28:00.508885 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:28:29.724329 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:29.724248 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:28:30.509883 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:28:30.509844 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:28:34.382267 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:34.382232 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr"] Oct 31 01:28:34.382748 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:34.382619 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kserve-container" containerID="cri-o://9087f570fc99872d4482731aaee56289b06c55a646a9500a2a3ad55cf1ca3ae3" gracePeriod=30 Oct 31 01:28:34.382748 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:34.382692 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kube-rbac-proxy" containerID="cri-o://eb8057b3b2e44b52ef04694acfa545951441096d7980bd6a1f7b0382849badc0" gracePeriod=30 Oct 31 01:28:34.382895 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:34.382821 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kserve-agent" containerID="cri-o://a5afe37433e2edf419a6d720bf954f8c94d83211d31f7f339a94315b9f7f1a54" gracePeriod=30 Oct 31 01:28:34.575726 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:34.575621 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused" Oct 31 01:28:34.864732 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:34.864697 5939 generic.go:356] "Generic (PLEG): container finished" podID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerID="eb8057b3b2e44b52ef04694acfa545951441096d7980bd6a1f7b0382849badc0" exitCode=0 Oct 31 01:28:34.864911 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:34.864752 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" event={"ID":"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b","Type":"ContainerDied","Data":"eb8057b3b2e44b52ef04694acfa545951441096d7980bd6a1f7b0382849badc0"} Oct 31 01:28:36.875440 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:36.875407 5939 generic.go:356] "Generic (PLEG): container finished" podID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerID="9087f570fc99872d4482731aaee56289b06c55a646a9500a2a3ad55cf1ca3ae3" exitCode=0 Oct 31 01:28:36.875818 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:36.875475 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" event={"ID":"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b","Type":"ContainerDied","Data":"9087f570fc99872d4482731aaee56289b06c55a646a9500a2a3ad55cf1ca3ae3"} Oct 31 01:28:39.576197 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:39.576148 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused" Oct 31 01:28:39.581734 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:39.581705 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.65:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.65:8080: connect: connection refused" Oct 31 01:28:44.575411 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:44.575364 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused" Oct 31 01:28:44.575876 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:44.575529 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:28:49.575604 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:49.575555 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused" Oct 31 01:28:49.582018 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:49.581990 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.65:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.65:8080: connect: connection refused" Oct 31 01:28:54.576110 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:54.576065 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused" Oct 31 01:28:59.575449 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:59.575399 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused" Oct 31 01:28:59.581870 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:59.581842 5939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.65:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.65:8080: connect: connection refused" Oct 31 01:28:59.581997 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:28:59.581984 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:29:00.508634 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:29:00.508595 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:29:04.556912 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.556884 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:29:04.719303 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.719194 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-cabundle-cert\") pod \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " Oct 31 01:29:04.719303 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.719312 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-proxy-tls\") pod \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " Oct 31 01:29:04.719563 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.719337 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kube-rbac-proxy-sar-config\") pod \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " Oct 31 01:29:04.719563 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.719387 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kserve-provision-location\") pod \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " Oct 31 01:29:04.719563 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.719412 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kube-api-access-b7ppj\" (UniqueName: \"kubernetes.io/projected/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kube-api-access-b7ppj\") pod \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\" (UID: \"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b\") " Oct 31 01:29:04.719764 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.719725 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-cabundle-cert" pod "90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" (UID: "90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:04.719764 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.719750 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kserve-provision-location" pod "90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" (UID: "90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:04.719852 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.719753 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kube-rbac-proxy-sar-config" pod "90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" (UID: "90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:04.721499 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.721479 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-proxy-tls" pod "90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" (UID: "90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:04.721606 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.721588 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kube-api-access-b7ppj" pod "90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" (UID: "90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b"). InnerVolumeSpecName "kube-api-access-b7ppj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:04.820926 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.820882 5939 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-proxy-tls\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:04.820926 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.820923 5939 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:04.820926 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.820937 5939 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kserve-provision-location\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:04.821164 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.820947 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7ppj\" (UniqueName: \"kubernetes.io/projected/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-kube-api-access-b7ppj\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:04.821164 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.820956 5939 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b-cabundle-cert\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:04.990586 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.990499 5939 generic.go:356] "Generic (PLEG): container finished" podID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerID="a5afe37433e2edf419a6d720bf954f8c94d83211d31f7f339a94315b9f7f1a54" exitCode=137 Oct 31 01:29:04.990586 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.990544 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" event={"ID":"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b","Type":"ContainerDied","Data":"a5afe37433e2edf419a6d720bf954f8c94d83211d31f7f339a94315b9f7f1a54"} Oct 31 01:29:04.990586 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.990582 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" Oct 31 01:29:04.990831 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.990613 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr" event={"ID":"90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b","Type":"ContainerDied","Data":"4b11dbbedb64e5523b4e0b29ade46317b1ebe735ae2335a4c6b9bba1eb42d23f"} Oct 31 01:29:04.990831 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.990657 5939 scope.go:122] "RemoveContainer" containerID="eb8057b3b2e44b52ef04694acfa545951441096d7980bd6a1f7b0382849badc0" Oct 31 01:29:04.999835 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:04.999817 5939 scope.go:122] "RemoveContainer" containerID="a5afe37433e2edf419a6d720bf954f8c94d83211d31f7f339a94315b9f7f1a54" Oct 31 01:29:05.007395 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:05.007373 5939 scope.go:122] "RemoveContainer" containerID="9087f570fc99872d4482731aaee56289b06c55a646a9500a2a3ad55cf1ca3ae3" Oct 31 01:29:05.014517 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:05.014500 5939 scope.go:122] "RemoveContainer" containerID="fcecd32799d1fadf0f3d9cbf3eb601bb91f5a73b8b88da3312df4d10547e8ca3" Oct 31 01:29:05.017394 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:05.017368 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr"] Oct 31 01:29:05.023047 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:05.023031 5939 scope.go:122] "RemoveContainer" containerID="eb8057b3b2e44b52ef04694acfa545951441096d7980bd6a1f7b0382849badc0" Oct 31 01:29:05.023332 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:29:05.023313 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb8057b3b2e44b52ef04694acfa545951441096d7980bd6a1f7b0382849badc0\": container with ID starting with eb8057b3b2e44b52ef04694acfa545951441096d7980bd6a1f7b0382849badc0 not found: ID does not exist" containerID="eb8057b3b2e44b52ef04694acfa545951441096d7980bd6a1f7b0382849badc0" Oct 31 01:29:05.023412 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:05.023341 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb8057b3b2e44b52ef04694acfa545951441096d7980bd6a1f7b0382849badc0"} err="failed to get container status \"eb8057b3b2e44b52ef04694acfa545951441096d7980bd6a1f7b0382849badc0\": rpc error: code = NotFound desc = could not find container \"eb8057b3b2e44b52ef04694acfa545951441096d7980bd6a1f7b0382849badc0\": container with ID starting with eb8057b3b2e44b52ef04694acfa545951441096d7980bd6a1f7b0382849badc0 not found: ID does not exist" Oct 31 01:29:05.023412 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:05.023358 5939 scope.go:122] "RemoveContainer" containerID="a5afe37433e2edf419a6d720bf954f8c94d83211d31f7f339a94315b9f7f1a54" Oct 31 01:29:05.023412 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:05.023365 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-f4cda2dafd/isvc-sklearn-mcp-predictor-79b8b57d96-8qmwr"] Oct 31 01:29:05.023609 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:29:05.023591 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5afe37433e2edf419a6d720bf954f8c94d83211d31f7f339a94315b9f7f1a54\": container with ID starting with a5afe37433e2edf419a6d720bf954f8c94d83211d31f7f339a94315b9f7f1a54 not found: ID does not exist" containerID="a5afe37433e2edf419a6d720bf954f8c94d83211d31f7f339a94315b9f7f1a54" Oct 31 01:29:05.023644 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:05.023613 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5afe37433e2edf419a6d720bf954f8c94d83211d31f7f339a94315b9f7f1a54"} err="failed to get container status \"a5afe37433e2edf419a6d720bf954f8c94d83211d31f7f339a94315b9f7f1a54\": rpc error: code = NotFound desc = could not find container \"a5afe37433e2edf419a6d720bf954f8c94d83211d31f7f339a94315b9f7f1a54\": container with ID starting with a5afe37433e2edf419a6d720bf954f8c94d83211d31f7f339a94315b9f7f1a54 not found: ID does not exist" Oct 31 01:29:05.023644 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:05.023628 5939 scope.go:122] "RemoveContainer" containerID="9087f570fc99872d4482731aaee56289b06c55a646a9500a2a3ad55cf1ca3ae3" Oct 31 01:29:05.023892 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:29:05.023875 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9087f570fc99872d4482731aaee56289b06c55a646a9500a2a3ad55cf1ca3ae3\": container with ID starting with 9087f570fc99872d4482731aaee56289b06c55a646a9500a2a3ad55cf1ca3ae3 not found: ID does not exist" containerID="9087f570fc99872d4482731aaee56289b06c55a646a9500a2a3ad55cf1ca3ae3" Oct 31 01:29:05.023938 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:05.023896 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9087f570fc99872d4482731aaee56289b06c55a646a9500a2a3ad55cf1ca3ae3"} err="failed to get container status \"9087f570fc99872d4482731aaee56289b06c55a646a9500a2a3ad55cf1ca3ae3\": rpc error: code = NotFound desc = could not find container \"9087f570fc99872d4482731aaee56289b06c55a646a9500a2a3ad55cf1ca3ae3\": container with ID starting with 9087f570fc99872d4482731aaee56289b06c55a646a9500a2a3ad55cf1ca3ae3 not found: ID does not exist" Oct 31 01:29:05.023938 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:05.023908 5939 scope.go:122] "RemoveContainer" containerID="fcecd32799d1fadf0f3d9cbf3eb601bb91f5a73b8b88da3312df4d10547e8ca3" Oct 31 01:29:05.024141 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:29:05.024127 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcecd32799d1fadf0f3d9cbf3eb601bb91f5a73b8b88da3312df4d10547e8ca3\": container with ID starting with fcecd32799d1fadf0f3d9cbf3eb601bb91f5a73b8b88da3312df4d10547e8ca3 not found: ID does not exist" containerID="fcecd32799d1fadf0f3d9cbf3eb601bb91f5a73b8b88da3312df4d10547e8ca3" Oct 31 01:29:05.024186 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:05.024144 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcecd32799d1fadf0f3d9cbf3eb601bb91f5a73b8b88da3312df4d10547e8ca3"} err="failed to get container status \"fcecd32799d1fadf0f3d9cbf3eb601bb91f5a73b8b88da3312df4d10547e8ca3\": rpc error: code = NotFound desc = could not find container \"fcecd32799d1fadf0f3d9cbf3eb601bb91f5a73b8b88da3312df4d10547e8ca3\": container with ID starting with fcecd32799d1fadf0f3d9cbf3eb601bb91f5a73b8b88da3312df4d10547e8ca3 not found: ID does not exist" Oct 31 01:29:05.652236 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:05.652204 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" path="/var/lib/kubelet/pods/90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b/volumes" Oct 31 01:29:30.509859 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:29:30.509817 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:29:48.059395 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:48.059350 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 01:29:48.064200 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:48.064180 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dhwx4_e7e79efa-c2ed-4cc0-928c-e0b9abf5275b/ovn-acl-logging/0.log" Oct 31 01:29:53.040877 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.040846 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qplp5/must-gather-7mrzl"] Oct 31 01:29:53.041264 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.041202 5939 cpu_manager.go:414] "containerMap: removing container" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="storage-initializer" Oct 31 01:29:53.041264 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.041214 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="storage-initializer" Oct 31 01:29:53.041264 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.041230 5939 cpu_manager.go:414] "containerMap: removing container" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kserve-container" Oct 31 01:29:53.041264 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.041236 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kserve-container" Oct 31 01:29:53.041264 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.041244 5939 cpu_manager.go:414] "containerMap: removing container" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kube-rbac-proxy" Oct 31 01:29:53.041264 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.041249 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kube-rbac-proxy" Oct 31 01:29:53.041264 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.041262 5939 cpu_manager.go:414] "containerMap: removing container" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kserve-agent" Oct 31 01:29:53.041264 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.041268 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kserve-agent" Oct 31 01:29:53.041524 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.041315 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kserve-container" Oct 31 01:29:53.041524 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.041322 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kube-rbac-proxy" Oct 31 01:29:53.041524 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.041329 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a635d7-7f4b-4dbb-b9cc-bc4df81ff47b" containerName="kserve-agent" Oct 31 01:29:53.048588 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.048560 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qplp5/must-gather-7mrzl" Oct 31 01:29:53.055349 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.055321 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qplp5\"/\"kube-root-ca.crt\"" Oct 31 01:29:53.055516 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.055396 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qplp5\"/\"openshift-service-ca.crt\"" Oct 31 01:29:53.055516 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.055454 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qplp5/must-gather-7mrzl"] Oct 31 01:29:53.055516 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.055457 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qplp5\"/\"default-dockercfg-ddmxw\"" Oct 31 01:29:53.155043 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.155006 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fbfdc8d9-464d-4762-b7ae-216464490306-must-gather-output\") pod \"must-gather-7mrzl\" (UID: \"fbfdc8d9-464d-4762-b7ae-216464490306\") " pod="openshift-must-gather-qplp5/must-gather-7mrzl" Oct 31 01:29:53.155219 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.155057 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwwcc\" (UniqueName: \"kubernetes.io/projected/fbfdc8d9-464d-4762-b7ae-216464490306-kube-api-access-rwwcc\") pod \"must-gather-7mrzl\" (UID: \"fbfdc8d9-464d-4762-b7ae-216464490306\") " pod="openshift-must-gather-qplp5/must-gather-7mrzl" Oct 31 01:29:53.255609 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.255568 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwwcc\" (UniqueName: \"kubernetes.io/projected/fbfdc8d9-464d-4762-b7ae-216464490306-kube-api-access-rwwcc\") pod \"must-gather-7mrzl\" (UID: \"fbfdc8d9-464d-4762-b7ae-216464490306\") " pod="openshift-must-gather-qplp5/must-gather-7mrzl" Oct 31 01:29:53.255943 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.255733 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fbfdc8d9-464d-4762-b7ae-216464490306-must-gather-output\") pod \"must-gather-7mrzl\" (UID: \"fbfdc8d9-464d-4762-b7ae-216464490306\") " pod="openshift-must-gather-qplp5/must-gather-7mrzl" Oct 31 01:29:53.256078 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.256058 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fbfdc8d9-464d-4762-b7ae-216464490306-must-gather-output\") pod \"must-gather-7mrzl\" (UID: \"fbfdc8d9-464d-4762-b7ae-216464490306\") " pod="openshift-must-gather-qplp5/must-gather-7mrzl" Oct 31 01:29:53.270239 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.270212 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwwcc\" (UniqueName: \"kubernetes.io/projected/fbfdc8d9-464d-4762-b7ae-216464490306-kube-api-access-rwwcc\") pod \"must-gather-7mrzl\" (UID: \"fbfdc8d9-464d-4762-b7ae-216464490306\") " pod="openshift-must-gather-qplp5/must-gather-7mrzl" Oct 31 01:29:53.378415 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.378380 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qplp5/must-gather-7mrzl" Oct 31 01:29:53.654106 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.654036 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qplp5/must-gather-7mrzl"] Oct 31 01:29:53.661507 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:53.661487 5939 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:29:54.176836 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:29:54.176802 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qplp5/must-gather-7mrzl" event={"ID":"fbfdc8d9-464d-4762-b7ae-216464490306","Type":"ContainerStarted","Data":"660bf1be6d9b97b8136b3867d368a33a8fcb3099d831ffe6f05dd44993dcc089"} Oct 31 01:30:00.210588 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:00.210550 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qplp5/must-gather-7mrzl" event={"ID":"fbfdc8d9-464d-4762-b7ae-216464490306","Type":"ContainerStarted","Data":"417fdb2cc705059dccdaba3797eda040ddcb91796a7f535df03a69b26af31e17"} Oct 31 01:30:00.210588 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:00.210594 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qplp5/must-gather-7mrzl" event={"ID":"fbfdc8d9-464d-4762-b7ae-216464490306","Type":"ContainerStarted","Data":"4381ca18768cb73a10be936eb978933baa4daeac39af1fd1c317bc586f0da773"} Oct 31 01:30:00.243043 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:00.242992 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qplp5/must-gather-7mrzl" podStartSLOduration=1.032954998 podStartE2EDuration="7.242979198s" totalImagesPullingTime="6.2100242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:00.242304255 +0000 UTC m=+3612.731579435" watchObservedRunningTime="2026-10-31 01:30:00.242979198 +0000 UTC m=+3612.732254376" Oct 31 01:30:00.513154 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:30:00.513058 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:30:20.305297 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:20.305205 5939 generic.go:356] "Generic (PLEG): container finished" podID="fbfdc8d9-464d-4762-b7ae-216464490306" containerID="4381ca18768cb73a10be936eb978933baa4daeac39af1fd1c317bc586f0da773" exitCode=0 Oct 31 01:30:20.305297 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:20.305280 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qplp5/must-gather-7mrzl" event={"ID":"fbfdc8d9-464d-4762-b7ae-216464490306","Type":"ContainerDied","Data":"4381ca18768cb73a10be936eb978933baa4daeac39af1fd1c317bc586f0da773"} Oct 31 01:30:20.305907 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:20.305732 5939 scope.go:122] "RemoveContainer" containerID="4381ca18768cb73a10be936eb978933baa4daeac39af1fd1c317bc586f0da773" Oct 31 01:30:21.113298 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:21.113267 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-qplp5_must-gather-7mrzl_fbfdc8d9-464d-4762-b7ae-216464490306/gather/0.log" Oct 31 01:30:25.452824 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:25.452793 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dvl2d_e7b81351-0c71-49b6-bd86-e48483b8b555/global-pull-secret-syncer/0.log" Oct 31 01:30:25.767200 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:25.767127 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b64jh_c892baf2-ce51-4c47-8d79-f75f4f50c09d/konnectivity-agent/0.log" Oct 31 01:30:25.954395 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:25.954359 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-213.us-east-2.compute.internal_2b347d22e86cd58c1b6312f0f07ef259/haproxy/0.log" Oct 31 01:30:26.645505 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:26.645474 5939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-qplp5/must-gather-7mrzl"] Oct 31 01:30:26.645941 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:26.645656 5939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-must-gather-qplp5/must-gather-7mrzl" podUID="fbfdc8d9-464d-4762-b7ae-216464490306" containerName="copy" containerID="cri-o://417fdb2cc705059dccdaba3797eda040ddcb91796a7f535df03a69b26af31e17" gracePeriod=2 Oct 31 01:30:26.652398 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:26.651930 5939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-qplp5/must-gather-7mrzl"] Oct 31 01:30:26.874474 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:26.874439 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-qplp5_must-gather-7mrzl_fbfdc8d9-464d-4762-b7ae-216464490306/copy/0.log" Oct 31 01:30:26.877080 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:26.877063 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qplp5/must-gather-7mrzl" Oct 31 01:30:26.879946 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:30:26.879924 5939 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-7mrzl\" is forbidden: User \"system:node:ip-10-0-143-213.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-qplp5\": no relationship found between node 'ip-10-0-143-213.us-east-2.compute.internal' and this object" podUID="fbfdc8d9-464d-4762-b7ae-216464490306" pod="openshift-must-gather-qplp5/must-gather-7mrzl" Oct 31 01:30:26.979745 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:26.979635 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbfdc8d9-464d-4762-b7ae-216464490306-must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fbfdc8d9-464d-4762-b7ae-216464490306-must-gather-output\") pod \"fbfdc8d9-464d-4762-b7ae-216464490306\" (UID: \"fbfdc8d9-464d-4762-b7ae-216464490306\") " Oct 31 01:30:26.979745 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:26.979737 5939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbfdc8d9-464d-4762-b7ae-216464490306-kube-api-access-rwwcc\" (UniqueName: \"kubernetes.io/projected/fbfdc8d9-464d-4762-b7ae-216464490306-kube-api-access-rwwcc\") pod \"fbfdc8d9-464d-4762-b7ae-216464490306\" (UID: \"fbfdc8d9-464d-4762-b7ae-216464490306\") " Oct 31 01:30:26.981050 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:26.981023 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfdc8d9-464d-4762-b7ae-216464490306-must-gather-output" pod "fbfdc8d9-464d-4762-b7ae-216464490306" (UID: "fbfdc8d9-464d-4762-b7ae-216464490306"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:26.981927 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:26.981903 5939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfdc8d9-464d-4762-b7ae-216464490306-kube-api-access-rwwcc" pod "fbfdc8d9-464d-4762-b7ae-216464490306" (UID: "fbfdc8d9-464d-4762-b7ae-216464490306"). InnerVolumeSpecName "kube-api-access-rwwcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:27.081017 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:27.080981 5939 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fbfdc8d9-464d-4762-b7ae-216464490306-must-gather-output\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:27.081017 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:27.081008 5939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwwcc\" (UniqueName: \"kubernetes.io/projected/fbfdc8d9-464d-4762-b7ae-216464490306-kube-api-access-rwwcc\") on node \"ip-10-0-143-213.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:27.334644 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:27.334572 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-qplp5_must-gather-7mrzl_fbfdc8d9-464d-4762-b7ae-216464490306/copy/0.log" Oct 31 01:30:27.334934 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:27.334910 5939 generic.go:356] "Generic (PLEG): container finished" podID="fbfdc8d9-464d-4762-b7ae-216464490306" containerID="417fdb2cc705059dccdaba3797eda040ddcb91796a7f535df03a69b26af31e17" exitCode=143 Oct 31 01:30:27.334987 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:27.334960 5939 scope.go:122] "RemoveContainer" containerID="417fdb2cc705059dccdaba3797eda040ddcb91796a7f535df03a69b26af31e17" Oct 31 01:30:27.335038 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:27.334960 5939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qplp5/must-gather-7mrzl" Oct 31 01:30:27.337949 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:30:27.337922 5939 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-7mrzl\" is forbidden: User \"system:node:ip-10-0-143-213.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-qplp5\": no relationship found between node 'ip-10-0-143-213.us-east-2.compute.internal' and this object" podUID="fbfdc8d9-464d-4762-b7ae-216464490306" pod="openshift-must-gather-qplp5/must-gather-7mrzl" Oct 31 01:30:27.343831 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:27.343808 5939 scope.go:122] "RemoveContainer" containerID="4381ca18768cb73a10be936eb978933baa4daeac39af1fd1c317bc586f0da773" Oct 31 01:30:27.346555 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:30:27.346519 5939 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-7mrzl\" is forbidden: User \"system:node:ip-10-0-143-213.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-qplp5\": no relationship found between node 'ip-10-0-143-213.us-east-2.compute.internal' and this object" podUID="fbfdc8d9-464d-4762-b7ae-216464490306" pod="openshift-must-gather-qplp5/must-gather-7mrzl" Oct 31 01:30:27.355813 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:27.355797 5939 scope.go:122] "RemoveContainer" containerID="417fdb2cc705059dccdaba3797eda040ddcb91796a7f535df03a69b26af31e17" Oct 31 01:30:27.356085 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:30:27.356068 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"417fdb2cc705059dccdaba3797eda040ddcb91796a7f535df03a69b26af31e17\": container with ID starting with 417fdb2cc705059dccdaba3797eda040ddcb91796a7f535df03a69b26af31e17 not found: ID does not exist" containerID="417fdb2cc705059dccdaba3797eda040ddcb91796a7f535df03a69b26af31e17" Oct 31 01:30:27.356133 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:27.356094 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"417fdb2cc705059dccdaba3797eda040ddcb91796a7f535df03a69b26af31e17"} err="failed to get container status \"417fdb2cc705059dccdaba3797eda040ddcb91796a7f535df03a69b26af31e17\": rpc error: code = NotFound desc = could not find container \"417fdb2cc705059dccdaba3797eda040ddcb91796a7f535df03a69b26af31e17\": container with ID starting with 417fdb2cc705059dccdaba3797eda040ddcb91796a7f535df03a69b26af31e17 not found: ID does not exist" Oct 31 01:30:27.356133 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:27.356111 5939 scope.go:122] "RemoveContainer" containerID="4381ca18768cb73a10be936eb978933baa4daeac39af1fd1c317bc586f0da773" Oct 31 01:30:27.356373 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:30:27.356355 5939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4381ca18768cb73a10be936eb978933baa4daeac39af1fd1c317bc586f0da773\": container with ID starting with 4381ca18768cb73a10be936eb978933baa4daeac39af1fd1c317bc586f0da773 not found: ID does not exist" containerID="4381ca18768cb73a10be936eb978933baa4daeac39af1fd1c317bc586f0da773" Oct 31 01:30:27.356427 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:27.356378 5939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4381ca18768cb73a10be936eb978933baa4daeac39af1fd1c317bc586f0da773"} err="failed to get container status \"4381ca18768cb73a10be936eb978933baa4daeac39af1fd1c317bc586f0da773\": rpc error: code = NotFound desc = could not find container \"4381ca18768cb73a10be936eb978933baa4daeac39af1fd1c317bc586f0da773\": container with ID starting with 4381ca18768cb73a10be936eb978933baa4daeac39af1fd1c317bc586f0da773 not found: ID does not exist" Oct 31 01:30:27.652549 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:30:27.652512 5939 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-7mrzl\" is forbidden: User \"system:node:ip-10-0-143-213.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-qplp5\": no relationship found between node 'ip-10-0-143-213.us-east-2.compute.internal' and this object" podUID="fbfdc8d9-464d-4762-b7ae-216464490306" pod="openshift-must-gather-qplp5/must-gather-7mrzl" Oct 31 01:30:27.653627 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:27.653591 5939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbfdc8d9-464d-4762-b7ae-216464490306" path="/var/lib/kubelet/pods/fbfdc8d9-464d-4762-b7ae-216464490306/volumes" Oct 31 01:30:30.509527 ip-10-0-143-213 kubenswrapper[5939]: E1031 01:30:30.509490 5939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w49pc" containerName="extractor" Oct 31 01:30:31.111814 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:31.111785 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4498092-a01a-4bad-b4a3-7197685a3272/alertmanager/0.log" Oct 31 01:30:31.155378 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:31.155345 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4498092-a01a-4bad-b4a3-7197685a3272/config-reloader/0.log" Oct 31 01:30:31.199289 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:31.199262 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4498092-a01a-4bad-b4a3-7197685a3272/kube-rbac-proxy-web/0.log" Oct 31 01:30:31.233255 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:31.233231 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4498092-a01a-4bad-b4a3-7197685a3272/kube-rbac-proxy/0.log" Oct 31 01:30:31.266477 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:31.266446 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4498092-a01a-4bad-b4a3-7197685a3272/kube-rbac-proxy-metric/0.log" Oct 31 01:30:31.300996 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:31.300972 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4498092-a01a-4bad-b4a3-7197685a3272/prom-label-proxy/0.log" Oct 31 01:30:31.337809 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:31.337785 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4498092-a01a-4bad-b4a3-7197685a3272/init-config-reloader/0.log" Oct 31 01:30:31.457901 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:31.457804 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mjkzk_8d06594d-071d-47bb-ade5-16298fd91e2c/kube-state-metrics/0.log" Oct 31 01:30:31.491409 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:31.491382 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mjkzk_8d06594d-071d-47bb-ade5-16298fd91e2c/kube-rbac-proxy-main/0.log" Oct 31 01:30:31.550288 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:31.550254 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mjkzk_8d06594d-071d-47bb-ade5-16298fd91e2c/kube-rbac-proxy-self/0.log" Oct 31 01:30:31.602332 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:31.602304 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-67cc7bf54c-zxwf4_b2eb2a42-2573-414d-8516-aac46f430550/metrics-server/0.log" Oct 31 01:30:31.639479 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:31.639445 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-65wl9_266a13d4-3f45-44d6-ac0e-3e171ce4e928/monitoring-plugin/0.log" Oct 31 01:30:31.931984 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:31.931962 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vl8l6_a4677743-13cb-4a3a-b1df-8bd6baef94f4/node-exporter/0.log" Oct 31 01:30:31.964797 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:31.964772 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vl8l6_a4677743-13cb-4a3a-b1df-8bd6baef94f4/kube-rbac-proxy/0.log" Oct 31 01:30:31.999201 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:31.999178 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vl8l6_a4677743-13cb-4a3a-b1df-8bd6baef94f4/init-textfile/0.log" Oct 31 01:30:32.039779 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.039748 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-srmg2_40c70642-d50d-4f92-8ca1-10015c1a40cd/kube-rbac-proxy-main/0.log" Oct 31 01:30:32.077068 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.077044 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-srmg2_40c70642-d50d-4f92-8ca1-10015c1a40cd/kube-rbac-proxy-self/0.log" Oct 31 01:30:32.114699 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.114677 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-srmg2_40c70642-d50d-4f92-8ca1-10015c1a40cd/openshift-state-metrics/0.log" Oct 31 01:30:32.163202 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.163162 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e9279c14-f83e-412e-acb9-6ec7859541f5/prometheus/0.log" Oct 31 01:30:32.194680 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.194589 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e9279c14-f83e-412e-acb9-6ec7859541f5/config-reloader/0.log" Oct 31 01:30:32.231295 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.231272 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e9279c14-f83e-412e-acb9-6ec7859541f5/thanos-sidecar/0.log" Oct 31 01:30:32.268362 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.268339 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e9279c14-f83e-412e-acb9-6ec7859541f5/kube-rbac-proxy-web/0.log" Oct 31 01:30:32.323507 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.323470 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e9279c14-f83e-412e-acb9-6ec7859541f5/kube-rbac-proxy/0.log" Oct 31 01:30:32.359338 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.359309 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e9279c14-f83e-412e-acb9-6ec7859541f5/kube-rbac-proxy-thanos/0.log" Oct 31 01:30:32.397177 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.397123 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e9279c14-f83e-412e-acb9-6ec7859541f5/init-config-reloader/0.log" Oct 31 01:30:32.555826 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.555734 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-766d564798-98prp_6a6e2db5-ca96-4448-8eb5-65e33c760582/telemeter-client/0.log" Oct 31 01:30:32.592724 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.592697 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-766d564798-98prp_6a6e2db5-ca96-4448-8eb5-65e33c760582/reload/0.log" Oct 31 01:30:32.626632 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.626604 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-766d564798-98prp_6a6e2db5-ca96-4448-8eb5-65e33c760582/kube-rbac-proxy/0.log" Oct 31 01:30:32.671073 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.671038 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bcdf957ff-s472c_86e13c77-adcc-4f59-b1a4-93d090e2379e/thanos-query/0.log" Oct 31 01:30:32.705273 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.705250 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bcdf957ff-s472c_86e13c77-adcc-4f59-b1a4-93d090e2379e/kube-rbac-proxy-web/0.log" Oct 31 01:30:32.745544 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.745508 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bcdf957ff-s472c_86e13c77-adcc-4f59-b1a4-93d090e2379e/kube-rbac-proxy/0.log" Oct 31 01:30:32.784345 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.784319 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bcdf957ff-s472c_86e13c77-adcc-4f59-b1a4-93d090e2379e/prom-label-proxy/0.log" Oct 31 01:30:32.819372 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.819281 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bcdf957ff-s472c_86e13c77-adcc-4f59-b1a4-93d090e2379e/kube-rbac-proxy-rules/0.log" Oct 31 01:30:32.854615 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:32.854580 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bcdf957ff-s472c_86e13c77-adcc-4f59-b1a4-93d090e2379e/kube-rbac-proxy-metrics/0.log" Oct 31 01:30:33.154100 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.154068 5939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb"] Oct 31 01:30:33.154417 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.154406 5939 cpu_manager.go:414] "containerMap: removing container" podUID="fbfdc8d9-464d-4762-b7ae-216464490306" containerName="copy" Oct 31 01:30:33.154461 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.154419 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbfdc8d9-464d-4762-b7ae-216464490306" containerName="copy" Oct 31 01:30:33.154461 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.154439 5939 cpu_manager.go:414] "containerMap: removing container" podUID="fbfdc8d9-464d-4762-b7ae-216464490306" containerName="gather" Oct 31 01:30:33.154461 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.154444 5939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbfdc8d9-464d-4762-b7ae-216464490306" containerName="gather" Oct 31 01:30:33.154557 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.154494 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbfdc8d9-464d-4762-b7ae-216464490306" containerName="copy" Oct 31 01:30:33.154557 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.154503 5939 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbfdc8d9-464d-4762-b7ae-216464490306" containerName="gather" Oct 31 01:30:33.159816 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.159793 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:33.172649 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.172625 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb"] Oct 31 01:30:33.232482 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.232452 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b31f613d-ee74-4d8f-a7ce-48b295ab3353-proc\") pod \"perf-node-gather-daemonset-txmdb\" (UID: \"b31f613d-ee74-4d8f-a7ce-48b295ab3353\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:33.232687 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.232495 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b31f613d-ee74-4d8f-a7ce-48b295ab3353-lib-modules\") pod \"perf-node-gather-daemonset-txmdb\" (UID: \"b31f613d-ee74-4d8f-a7ce-48b295ab3353\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:33.232687 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.232586 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f54cp\" (UniqueName: \"kubernetes.io/projected/b31f613d-ee74-4d8f-a7ce-48b295ab3353-kube-api-access-f54cp\") pod \"perf-node-gather-daemonset-txmdb\" (UID: \"b31f613d-ee74-4d8f-a7ce-48b295ab3353\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:33.232687 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.232626 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b31f613d-ee74-4d8f-a7ce-48b295ab3353-sys\") pod \"perf-node-gather-daemonset-txmdb\" (UID: \"b31f613d-ee74-4d8f-a7ce-48b295ab3353\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:33.232687 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.232650 5939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b31f613d-ee74-4d8f-a7ce-48b295ab3353-podres\") pod \"perf-node-gather-daemonset-txmdb\" (UID: \"b31f613d-ee74-4d8f-a7ce-48b295ab3353\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:33.257719 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.257694 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-whtv4\"/\"kube-root-ca.crt\"" Oct 31 01:30:33.257904 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.257837 5939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-whtv4\"/\"openshift-service-ca.crt\"" Oct 31 01:30:33.257904 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.257877 5939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-whtv4\"/\"default-dockercfg-94wtk\"" Oct 31 01:30:33.333370 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.333323 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b31f613d-ee74-4d8f-a7ce-48b295ab3353-proc\") pod \"perf-node-gather-daemonset-txmdb\" (UID: \"b31f613d-ee74-4d8f-a7ce-48b295ab3353\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:33.333370 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.333384 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b31f613d-ee74-4d8f-a7ce-48b295ab3353-lib-modules\") pod \"perf-node-gather-daemonset-txmdb\" (UID: \"b31f613d-ee74-4d8f-a7ce-48b295ab3353\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:33.333604 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.333403 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f54cp\" (UniqueName: \"kubernetes.io/projected/b31f613d-ee74-4d8f-a7ce-48b295ab3353-kube-api-access-f54cp\") pod \"perf-node-gather-daemonset-txmdb\" (UID: \"b31f613d-ee74-4d8f-a7ce-48b295ab3353\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:33.333604 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.333439 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b31f613d-ee74-4d8f-a7ce-48b295ab3353-sys\") pod \"perf-node-gather-daemonset-txmdb\" (UID: \"b31f613d-ee74-4d8f-a7ce-48b295ab3353\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:33.333604 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.333456 5939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b31f613d-ee74-4d8f-a7ce-48b295ab3353-podres\") pod \"perf-node-gather-daemonset-txmdb\" (UID: \"b31f613d-ee74-4d8f-a7ce-48b295ab3353\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:33.333604 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.333457 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b31f613d-ee74-4d8f-a7ce-48b295ab3353-proc\") pod \"perf-node-gather-daemonset-txmdb\" (UID: \"b31f613d-ee74-4d8f-a7ce-48b295ab3353\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:33.333604 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.333472 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b31f613d-ee74-4d8f-a7ce-48b295ab3353-lib-modules\") pod \"perf-node-gather-daemonset-txmdb\" (UID: \"b31f613d-ee74-4d8f-a7ce-48b295ab3353\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:33.333604 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.333528 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b31f613d-ee74-4d8f-a7ce-48b295ab3353-sys\") pod \"perf-node-gather-daemonset-txmdb\" (UID: \"b31f613d-ee74-4d8f-a7ce-48b295ab3353\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:33.333875 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.333625 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b31f613d-ee74-4d8f-a7ce-48b295ab3353-podres\") pod \"perf-node-gather-daemonset-txmdb\" (UID: \"b31f613d-ee74-4d8f-a7ce-48b295ab3353\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:33.348616 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.348591 5939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f54cp\" (UniqueName: \"kubernetes.io/projected/b31f613d-ee74-4d8f-a7ce-48b295ab3353-kube-api-access-f54cp\") pod \"perf-node-gather-daemonset-txmdb\" (UID: \"b31f613d-ee74-4d8f-a7ce-48b295ab3353\") " pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:33.472885 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.472794 5939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:33.751082 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:33.750994 5939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb"] Oct 31 01:30:33.754515 ip-10-0-143-213 kubenswrapper[5939]: W1031 01:30:33.754486 5939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb31f613d_ee74_4d8f_a7ce_48b295ab3353.slice/crio-567b65e1375b43a9aa931516d2ca23082110b333b480be09c94acf903be08a0d WatchSource:0}: Error finding container 567b65e1375b43a9aa931516d2ca23082110b333b480be09c94acf903be08a0d: Status 404 returned error can't find the container with id 567b65e1375b43a9aa931516d2ca23082110b333b480be09c94acf903be08a0d Oct 31 01:30:34.364421 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:34.364383 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" event={"ID":"b31f613d-ee74-4d8f-a7ce-48b295ab3353","Type":"ContainerStarted","Data":"e0a8de460b0b5b76e30a14176448885f9296e1e996cbd209e9c15ffb21ac9e81"} Oct 31 01:30:34.364600 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:34.364436 5939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" event={"ID":"b31f613d-ee74-4d8f-a7ce-48b295ab3353","Type":"ContainerStarted","Data":"567b65e1375b43a9aa931516d2ca23082110b333b480be09c94acf903be08a0d"} Oct 31 01:30:34.364600 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:34.364517 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:34.391130 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:34.391075 5939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" podStartSLOduration=1.391059606 podStartE2EDuration="1.391059606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:34.390938661 +0000 UTC m=+3646.880213840" watchObservedRunningTime="2026-10-31 01:30:34.391059606 +0000 UTC m=+3646.880334786" Oct 31 01:30:36.952083 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:36.952052 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xkqrm_37dd17e5-ba62-4f95-bfc7-05b132104e44/dns/0.log" Oct 31 01:30:36.986169 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:36.986146 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xkqrm_37dd17e5-ba62-4f95-bfc7-05b132104e44/kube-rbac-proxy/0.log" Oct 31 01:30:37.099637 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:37.099609 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wvff6_f94485d7-b1a8-4caa-bfac-39c7cf29bfaa/dns-node-resolver/0.log" Oct 31 01:30:37.733173 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:37.733141 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9l52l_79b89acd-a759-4036-bd91-16ee9ad9e288/node-ca/0.log" Oct 31 01:30:39.240236 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:39.240204 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pnmtp_57decfb3-58bb-4b9c-b834-205594071bdf/serve-healthcheck-canary/0.log" Oct 31 01:30:39.943507 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:39.943476 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w49pc_51dbc54b-218b-4274-9ebb-4cdbab400d22/kube-rbac-proxy/0.log" Oct 31 01:30:39.985973 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:39.985943 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w49pc_51dbc54b-218b-4274-9ebb-4cdbab400d22/exporter/0.log" Oct 31 01:30:40.021292 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:40.021266 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w49pc_51dbc54b-218b-4274-9ebb-4cdbab400d22/extractor/0.log" Oct 31 01:30:40.378686 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:40.378646 5939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-whtv4/perf-node-gather-daemonset-txmdb" Oct 31 01:30:44.055239 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:44.055205 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-mgjt5_266dfcf3-3908-4a9c-bb3c-c45b9407bc0c/s3-init/0.log" Oct 31 01:30:44.080581 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:44.080556 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-mgjt5_266dfcf3-3908-4a9c-bb3c-c45b9407bc0c/download-hf-model/0.log" Oct 31 01:30:44.137732 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:44.137704 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-25gmd_8f54ff05-0753-4dfd-a703-c1743e79458e/s3-tls-init-custom/0.log" Oct 31 01:30:44.200772 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:44.200735 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-25gmd_8f54ff05-0753-4dfd-a703-c1743e79458e/download-hf-model/0.log" Oct 31 01:30:44.237051 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:44.237027 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-9qhsg_97cda6ca-3db4-46be-ba8f-fb77be8db868/s3-tls-init-serving/0.log" Oct 31 01:30:44.270868 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:44.270843 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-9qhsg_97cda6ca-3db4-46be-ba8f-fb77be8db868/download-hf-model/0.log" Oct 31 01:30:44.355897 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:44.355820 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-685c9f986c-8lt65_23f1d2f3-affb-4340-8ed8-aa65b185f9da/seaweedfs-tls-custom/0.log" Oct 31 01:30:44.391807 ip-10-0-143-213 kubenswrapper[5939]: I1031 01:30:44.391777 5939 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-5c74974785-7qnl4_0489b906-3432-46a6-9c0b-2ef27f65d8fb/seaweedfs-tls-serving/0.log"