Oct 30 18:35:35.122188 ip-10-0-135-66 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 18:35:35.122199 ip-10-0-135-66 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 18:35:35.122206 ip-10-0-135-66 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 18:35:35.122455 ip-10-0-135-66 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 18:35:45.322385 ip-10-0-135-66 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 18:35:45.322399 ip-10-0-135-66 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c9ca2a92618e44afb46017642c895cce --
Oct 30 18:37:44.038904 ip-10-0-135-66 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:37:44.410715 ip-10-0-135-66 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:37:44.410715 ip-10-0-135-66 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:37:44.410715 ip-10-0-135-66 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:37:44.410715 ip-10-0-135-66 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:37:44.414555 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414472    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:44.414555 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414555    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:44.414555 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414559    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414563    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414566    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414569    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414572    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414574    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414577    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414579    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414582    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414585    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414587    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414590    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414593    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414595    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414599    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414604    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414608    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414610    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414613    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:44.414647 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414616    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414619    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414622    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414625    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414628    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414630    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414640    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414643    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414646    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414648    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414651    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414654    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414657    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414660    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414662    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414665    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414667    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414670    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414672    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:44.415083 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414675    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414677    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414680    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414682    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414684    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414687    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414689    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414692    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414694    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414696    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414699    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414701    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414704    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414707    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414709    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414712    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414715    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414718    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414721    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:44.415537 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414723    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414725    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414728    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414730    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414733    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414736    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414738    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414741    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414743    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414746    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414748    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414751    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414753    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414755    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414758    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414760    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414763    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414766    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414768    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414770    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:44.416010 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414773    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414776    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414778    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414781    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414784    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414786    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414789    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414791    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414794    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414798    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414800    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414803    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414805    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414808    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414810    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414813    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414815    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414820    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414822    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414825    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:44.416510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414828    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414830    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414833    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414836    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414840    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414845    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.414848    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415698    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415706    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415709    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415712    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415715    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415718    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415721    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415724    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415726    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415729    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415732    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415735    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415737    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:44.416967 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415740    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415743    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415746    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415749    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415752    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415754    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415757    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415759    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415763    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415767    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415770    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415773    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415776    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415779    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415782    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415785    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415788    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415792    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415795    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415798    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:44.417474 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415801    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415804    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415806    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415809    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415811    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415814    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415817    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415821    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415825    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415828    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415831    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415835    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415839    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415841    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415844    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415847    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415849    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415852    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415854    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:44.417941 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415857    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415859    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415862    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415864    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415867    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415870    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415872    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415874    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415877    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415879    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415882    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415885    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415887    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415890    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415892    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415895    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415897    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415899    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415902    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:44.418397 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415904    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415907    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415910    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415913    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415915    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415918    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415922    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415924    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415927    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415930    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415932    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415935    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415938    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415940    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415942    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415945    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415947    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415950    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415952    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:44.418876 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415955    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415957    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415960    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415962    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415965    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415968    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415970    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415973    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415976    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415978    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415981    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415983    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415986    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415988    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415991    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.415993    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416514    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416520    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416523    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416527    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:44.419366 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416530    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416533    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416536    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416538    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416541    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416544    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416546    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416549    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416551    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416554    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416557    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416559    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416562    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416565    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416567    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416570    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416573    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416575    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416578    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416581    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:44.419903 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416583    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416586    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416588    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416591    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416594    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416596    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416599    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416601    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416604    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416608    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416612    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416616    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416618    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416621    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416623    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416626    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416628    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416631    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416634    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:44.420456 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416636    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416639    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416641    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416644    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416646    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416649    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416651    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416655    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416657    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416660    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416662    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416665    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416667    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416669    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416672    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416674    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416677    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416680    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416682    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:44.420963 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416685    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416687    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416689    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416692    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416695    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416698    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416700    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416702    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416705    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416707    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416710    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416712    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416717    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416720    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416722    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416726    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416729    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416732    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416734    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:44.421607 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416737    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416739    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416741    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416744    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416746    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416749    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416751    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416754    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416756    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416759    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416761    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416764    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416766    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416769    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416771    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416774    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416776    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416780    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416782    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416786    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:44.422135 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416788    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416791    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416793    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416795    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.416798    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416888    2575 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416895    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416901    2575 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416906    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416910    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416913    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416918    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416922    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416926    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416929    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416932    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416935    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416938    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416941    2575 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416944    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416947    2575 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416950    2575 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416953    2575 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:37:44.422668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416958    2575 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416961    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416964    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416967    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416971    2575 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416975    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416980    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416984    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416987    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416990    2575 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416993    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416996    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.416999    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417002    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417010    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417013    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417016    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417019    2575 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417021    2575 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417025    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417029    2575 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417032    2575 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417035    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417038    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:37:44.423356 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417041    2575 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417045    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417048    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417051    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417054    2575 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417057    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417061    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417064    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417067    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417070    2575 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417073    2575 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417075    2575 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417080    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417083    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417087    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417090    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417093    2575 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417096    2575 flags.go:64] FLAG: --help="false"
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417099    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417103    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417106    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417109    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417113    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:37:44.424455 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417117    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417120    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417122    2575 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417149    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417153    2575 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417156    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417159    2575 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417162    2575 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417165    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417168    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417171    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417173    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417176    2575 flags.go:64] FLAG: --lock-file=""
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417179    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417182    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417185    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417191    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417194    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417197    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417200    2575 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417203    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417206    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417209    2575 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417214    2575 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417218    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:37:44.425459 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417221    2575 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417226    2575 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417229    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417232    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417235    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417238    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417242    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417245    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417248    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417255    2575 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417258    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417261    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417264    2575 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417267    2575 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417269    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417272    2575 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417275    2575 flags.go:64] FLAG: --port="10250"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417278    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417281    2575 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-04af32871652ebe57"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417285    2575 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417287    2575 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417290    2575 flags.go:64] FLAG: --register-node="true"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417293    2575 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417297    2575 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417300    2575 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417303    2575 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:37:44.426526 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417306    2575 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417310    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417313    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417317    2575 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417321    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417324    2575 flags.go:64] FLAG: --runonce="false"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417327    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417330    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417334    2575 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417337    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417340    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417343    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417347    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417351    2575 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417354    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417356    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417359    2575 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417362    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417366    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417368    2575 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417371    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417376    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417379    2575 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417382    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417386    2575 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:37:44.427664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417389    2575 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417392    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417395    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417398    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417401    2575 flags.go:64] FLAG: --v="2"
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417405    2575 flags.go:64] FLAG: --version="false"
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417409    2575 flags.go:64] FLAG: --vmodule=""
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417413    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417416    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417556    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417560    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417565    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417567    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417570    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417572    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417575    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417578    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417581    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417583    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417587    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417590    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:44.428522 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417592    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417596    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417600    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417603    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417606    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417609    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417612    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417614    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417617    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417619    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417622    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417624    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417627    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417629    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417632    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417634    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417637    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417639    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417642    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:44.429299 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417644    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417647    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417649    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417653    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417656    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417658    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417661    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417663    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417666    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417669    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417671    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417675    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417678    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417680    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417683    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417685    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417688    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417690    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417693    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:44.429954 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417695    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417698    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417700    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417703    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417705    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417708    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417710    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417713    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417715    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417718    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417720    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417723    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417726    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417728    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417731    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417733    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417737    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417740    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417742    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417745    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:44.430538 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417747    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417750    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417752    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417755    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417759    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417761    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417764    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417766    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417769    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417772    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417774    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417776    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417779    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417781    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417783    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417786    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417790    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417793    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417796    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:44.431039 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417799    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:44.431510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417802    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:44.431510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417804    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:44.431510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417807    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:44.431510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417809    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:44.431510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417812    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:44.431510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417815    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:44.431510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417817    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:44.431510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417820    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:44.431510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417824    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:44.431510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417826    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:44.431510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417829    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:44.431510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417831    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:44.431510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417834    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:44.431510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417837    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:44.431510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417839    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:44.431510 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417842    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:44.431878 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.417869    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:37:44.431878 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417905    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:44.431878 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417910    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:44.431878 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417916    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:44.431878 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417919    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:44.431878 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417923    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:44.431878 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417926    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:44.431878 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417929    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:44.431878 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417932    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:44.431878 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417934    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:44.431878 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417937    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:44.431878 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417940    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:44.431878 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417943    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:44.431878 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417945    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:44.431878 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417948    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:44.431878 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417951    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417953    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417956    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417959    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417961    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417964    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417967    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417969    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417972    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417975    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417978    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417981    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417984    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417986    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417989    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417991    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417994    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.417996    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418000    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:44.432274 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418003    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418005    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418008    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418011    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418013    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418015    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418018    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418020    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418023    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418025    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418027    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418030    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418032    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418035    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418038    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418040    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418043    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418045    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418048    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418050    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:44.432710 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418053    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418055    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418059    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418062    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418064    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418067    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418069    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418072    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418074    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418076    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418079    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418083    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418086    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418089    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418093    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418095    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418098    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418100    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418103    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418105    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:44.433278 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418108    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418112    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418115    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418118    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418121    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418136    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418139    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418142    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418144    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418147    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418150    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418153    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418156    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418159    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418163    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418165    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418168    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418170    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418173    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418175    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:44.433746 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418178    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418180    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418183    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418186    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418189    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418192    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418195    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418198    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418200    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418204    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418206    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418209    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.418211    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.425581    2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.425601    2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.425624    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.425632    2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425717    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425726    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:44.434229 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425732    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425737    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425743    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425747    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425752    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425756    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425760    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425764    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425768    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425775    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425780    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425784    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425788    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425793    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425797    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425801    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425806    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425809    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425814    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:44.434677 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425819    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425823    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425827    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425831    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425836    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425840    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425844    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425848    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425855    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425860    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425865    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425869    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425873    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425878    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425882    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425886    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425891    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425895    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425899    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425903    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:44.435115 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425908    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425912    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425916    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425921    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425925    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425929    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425934    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425938    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425942    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425947    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425951    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425955    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425960    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425964    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425969    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425974    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425978    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425982    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425989    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:44.435596 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.425996    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426002    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426009    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426016    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426021    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426026    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426031    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426036    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426041    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426045    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426049    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426053    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426057    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426062    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426066    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426070    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426074    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426078    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426083    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426087    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:44.436148 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426091    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426095    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426099    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426103    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426108    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426113    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426118    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426122    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426145    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426149    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426154    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426158    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426162    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426166    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426170    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426175    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426180    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426184    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426189    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426193    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:44.436628 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426197    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:44.437253 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426201    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:44.437253 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426206    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:44.437253 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426210    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:44.437253 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426214    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:44.437253 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426219    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:44.437253 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.426259    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:37:44.437253 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426483    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:44.437253 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426491    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:44.437253 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426496    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:44.437253 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426500    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:44.437253 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426504    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:44.437253 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426509    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:44.437253 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426513    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:44.437253 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426517    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:44.437253 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426521    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:44.437253 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426525    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426529    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426534    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426538    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426542    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426546    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426550    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426554    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426558    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426563    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426567    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426572    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426577    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426581    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426586    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426589    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426594    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426601    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426606    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:44.437679 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426610    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426614    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426619    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426623    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426630    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426635    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426642    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426647    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426652    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426657    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426662    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426667    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426672    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426677    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426682    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426686    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426691    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426695    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426699    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:44.438118 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426703    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426708    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426712    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426716    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426720    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426724    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426728    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426733    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426737    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426742    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426746    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426750    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426754    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426758    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426762    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426767    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426771    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426774    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426779    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426785    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:44.438618 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426789    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426793    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426798    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426803    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426807    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426811    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426815    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426820    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426824    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426828    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426832    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426837    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426841    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426845    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426849    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426853    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426857    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426861    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426865    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426869    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:44.439168 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426874    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426878    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426882    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426887    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426891    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426895    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426898    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426902    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426907    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426911    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426915    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426919    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426923    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426927    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426931    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426935    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426940    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426944    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:44.439671 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.426947    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:44.440106 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.426985    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:37:44.440106 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427036    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:44.440106 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427045    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:44.440106 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427050    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:44.440106 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427054    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:44.440106 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427058    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:44.440106 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427062    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:44.440106 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427066    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:44.440106 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427070    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:44.440106 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427075    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:44.440106 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427079    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:44.440106 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427083    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:44.440106 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427087    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:44.440106 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427091    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:44.440106 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427096    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:44.440106 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427100    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427105    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427110    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427114    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427118    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427122    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427145    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427149    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427153    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427157    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427161    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427165    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427170    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427174    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427178    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427182    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427187    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427191    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427196    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427200    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:44.440610 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427204    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427208    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427212    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427218    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427222    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427226    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427230    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427234    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427239    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427243    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427247    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427251    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427256    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427260    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427265    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427269    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427273    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427277    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427281    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427285    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:44.441150 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427289    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427293    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427297    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427301    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427306    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427310    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427314    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427318    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427322    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427327    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427331    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427335    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427339    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427343    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427348    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427352    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427356    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427360    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427364    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427368    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:44.441619 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427372    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427377    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427381    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427385    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427389    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427394    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427398    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427402    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427407    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427411    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427415    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427419    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427423    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427427    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427431    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427435    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427439    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427443    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427448    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:44.442173 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427451    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427455    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427460    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427464    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427468    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427472    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427476    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427482    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427488    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427493    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427498    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427504    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:44.427508    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.427595    2575 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.431678    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.432508    2575 server.go:1014] "Starting client certificate rotation"
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.432600    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.432632    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:37:44.442692 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.434485    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:37:44.443174 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.435675    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:37:44.443518 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.443501    2575 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:37:44.446363 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.446348    2575 log.go:25] "Validated CRI v1 image API"
Oct 30 18:37:44.447969 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.447940    2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:37:44.452907 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.452882    2575 fs.go:135] Filesystem UUIDs: map[50f16d56-ad77-4ad8-82bc-346a2e08680a:/dev/nvme0n1p3 51d14c8d-fdf7-4b84-a193-a00a66fe12de:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 18:37:44.452964 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.452908    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 18:37:44.458711 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.458593    2575 manager.go:217] Machine: {Timestamp:2026-10-30 18:37:44.457248217 +0000 UTC m=+0.308282108 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2125824 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec237ca8c704693c9d8e1ef58fd1cce6 SystemUUID:ec237ca8-c704-693c-9d8e-1ef58fd1cce6 BootID:c9ca2a92-618e-44af-b460-17642c895cce Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:e0:f5:e3:ed Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:e0:f5:e3:ed Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:75:e8:b2:f6:f1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:37:44.458711 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.458710    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:37:44.458824 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.458792    2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:37:44.461814 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.461788    2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:37:44.461991 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.461816    2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-66.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:37:44.462080 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.461996    2575 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:37:44.462080 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.462009    2575 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:37:44.462080 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.462053    2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:37:44.462080 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.462076    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:37:44.462804 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.462792    2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:37:44.465481 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.465467    2575 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:37:44.465636 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.465625    2575 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:37:44.466244 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.466234    2575 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:37:44.466305 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.466250    2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:37:44.466305 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.466264    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:37:44.466305 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.466277    2575 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:37:44.466305 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.466289    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:37:44.466678 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.466664    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:37:44.469044 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.469005    2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:37:44.474594 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.474578    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:37:44.475353 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.475341    2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:37:44.475391 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.475372    2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:37:44.475548 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.475537    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:37:44.475588 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.475555    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:37:44.475588 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.475561    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:37:44.475588 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.475567    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:37:44.475588 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.475573    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:37:44.475588 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.475580    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:37:44.476023 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.476014    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:37:44.476058 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.476024    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:37:44.476058 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.476031    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:37:44.476058 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.476037    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:37:44.476058 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.476046    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:37:44.476058 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.476055    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:37:44.477225 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.477214    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:37:44.477263 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.477227    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:37:44.481047 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.481032    2575 server.go:1263] "Started kubelet"
Oct 30 18:37:44.481183 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.481107    2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:37:44.481249 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.481150    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:37:44.481249 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.481231    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:37:44.482026 ip-10-0-135-66 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:37:44.482480 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.482344    2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:37:44.483556 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.483544    2575 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:37:44.489308 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.489287    2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:37:44.489398 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.489292    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:37:44.489398 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:44.489304    2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 18:37:44.489516 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:44.489407    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found"
Oct 30 18:37:44.490213 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.490194    2575 factory.go:55] Registering systemd factory
Oct 30 18:37:44.490213 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.490209    2575 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:37:44.490473 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.490455    2575 factory.go:153] Registering CRI-O factory
Oct 30 18:37:44.490473 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.490472    2575 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:37:44.490629 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.490523    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:37:44.490629 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.490555    2575 factory.go:103] Registering Raw factory
Oct 30 18:37:44.490629 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.490573    2575 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:37:44.490979 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.490761    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:37:44.491030 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.490766    2575 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:37:44.491030 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.490998    2575 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:37:44.491116 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.491098    2575 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:37:44.491116 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.491112    2575 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:37:44.491522 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.491505    2575 manager.go:319] Starting recovery of all containers
Oct 30 18:37:44.503222 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.503201    2575 manager.go:324] Recovery completed
Oct 30 18:37:44.508425 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.508408    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:44.510155 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.510118    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:44.510207 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.510172    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:44.510207 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.510182    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:44.510745 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.510729    2575 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:37:44.510745 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.510741    2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:37:44.510808 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.510758    2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:37:44.510808 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.510730    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:44.511493 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.511477    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:44.511537 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.511507    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:44.511537 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.511517    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:44.511916 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.511905    2575 policy_none.go:50] "Start"
Oct 30 18:37:44.511947 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.511929    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:37:44.511947 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.511939    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:37:44.512922 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.512912    2575 policy_none.go:44] "Start"
Oct 30 18:37:44.556833 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.556790    2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:37:44.561321 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.561305    2575 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:37:44.561774 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:44.561759    2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:37:44.561810 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.561777    2575 server.go:87] "Starting device plugin registration server"
Oct 30 18:37:44.561995 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.561985    2575 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:37:44.562025 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.562006    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:37:44.562228 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.562209    2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:37:44.562322 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.562306    2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:37:44.562462 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.562441    2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:37:44.564060 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:44.564042    2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:37:44.564144 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:44.564093    2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-66.us-west-2.compute.internal\" not found"
Oct 30 18:37:44.620375 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.620350    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-66.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:44.620375 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:44.620358    2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-66.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 18:37:44.636476 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.636443    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:37:44.637905 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.637887    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:37:44.637985 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.637918    2575 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:37:44.637985 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.637943    2575 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:37:44.637985 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:44.637978    2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 18:37:44.662140 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.662065    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:44.663070 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.663056    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:44.663147 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.663084    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:44.663147 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.663106    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:44.663224 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.663155    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:44.667544 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:44.666461    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-66.us-west-2.compute.internal.18e3629403bd36e8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-66.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-66.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:37:44.48100324 +0000 UTC m=+0.332037128,LastTimestamp:2026-10-30 18:37:44.48100324 +0000 UTC m=+0.332037128,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-66.us-west-2.compute.internal,}"
Oct 30 18:37:44.738237 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.738202    2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal"]
Oct 30 18:37:44.738389 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.738268    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:44.739323 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.739308    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:44.739390 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.739337    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:44.739390 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.739347    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:44.740727 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.740714    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:44.740883 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.740866    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:44.740919 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.740901    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:44.741605 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.741577    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:44.741702 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.741610    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:44.741702 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.741625    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:44.741702 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.741632    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:44.741702 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.741659    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:44.741702 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.741673    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:44.742629 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.742613    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:44.742706 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.742643    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:44.743418 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.743401    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:44.743418 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.743421    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:44.743550 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.743429    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:44.763984 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.763964    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:37:44.764065 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.764009    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jzwl5"
Oct 30 18:37:44.771904 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:44.771876    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-66.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 18:37:44.772012 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:44.771965    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 18:37:44.772203 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:44.772098    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-66.us-west-2.compute.internal.18e36294057a0e9d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-66.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-66.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-66.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:37:44.510156445 +0000 UTC m=+0.361190335,LastTimestamp:2026-10-30 18:37:44.510156445 +0000 UTC m=+0.361190335,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-66.us-west-2.compute.internal,}"
Oct 30 18:37:44.772318 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:44.772237    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 18:37:44.772856 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:44.772839    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found" node="ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:44.776906 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:44.776881    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found" node="ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:44.792235 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.792201    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b5fcda9dfeca1f860cdbf0f9fc64e83-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"0b5fcda9dfeca1f860cdbf0f9fc64e83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:44.792235 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.792241    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b5fcda9dfeca1f860cdbf0f9fc64e83-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"0b5fcda9dfeca1f860cdbf0f9fc64e83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:44.792398 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.792257    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/80e81f6072b8791c05f541935e51a65e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"80e81f6072b8791c05f541935e51a65e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:44.830027 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.830004    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:44.830680 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:44.830664    2575 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-66.us-west-2.compute.internal\" not found" node="ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:44.892909 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.892879    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b5fcda9dfeca1f860cdbf0f9fc64e83-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"0b5fcda9dfeca1f860cdbf0f9fc64e83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:44.892909 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.892910    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b5fcda9dfeca1f860cdbf0f9fc64e83-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"0b5fcda9dfeca1f860cdbf0f9fc64e83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:44.893095 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.892927    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/80e81f6072b8791c05f541935e51a65e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"80e81f6072b8791c05f541935e51a65e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:44.893095 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.892963    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/80e81f6072b8791c05f541935e51a65e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"80e81f6072b8791c05f541935e51a65e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:44.893095 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.892980    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b5fcda9dfeca1f860cdbf0f9fc64e83-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"0b5fcda9dfeca1f860cdbf0f9fc64e83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:44.893095 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:44.892982    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b5fcda9dfeca1f860cdbf0f9fc64e83-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"0b5fcda9dfeca1f860cdbf0f9fc64e83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:44.928766 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:44.928708    2575 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-66.us-west-2.compute.internal\" not found" node="ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:45.031434 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:45.031409    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:45.032390 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:45.032374    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:45.032491 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:45.032402    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:45.032491 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:45.032412    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:45.032491 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:45.032443    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:45.075780 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:45.075761    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:45.080254 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:45.080239    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:45.091956 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:45.091940    2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:45.092027 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:45.091962    2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-66.us-west-2.compute.internal\": node \"ip-10-0-135-66.us-west-2.compute.internal\" not found"
Oct 30 18:37:45.207905 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:45.207836    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found"
Oct 30 18:37:45.308226 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:45.308194    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found"
Oct 30 18:37:45.409344 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:45.409309    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found"
Oct 30 18:37:45.432664 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:45.432641    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 18:37:45.433328 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:45.432827    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 18:37:45.433328 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:45.432802    2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a97600e569e5c4a9eafb16c9ff6ff0f2-36f3c3322d66c93d.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-135-66.us-west-2.compute.internal.18e36294057a0e9d\": read tcp 10.0.135.66:53774->13.184.142.95:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-135-66.us-west-2.compute.internal.18e36294057a0e9d  default   4759 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-66.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-66.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-66.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:37:44 +0000 UTC,LastTimestamp:2026-10-30 18:37:44.739323506 +0000 UTC m=+0.590357393,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-66.us-west-2.compute.internal,}"
Oct 30 18:37:45.509477 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:45.509417    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found"
Oct 30 18:37:45.610189 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:45.610163    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found"
Oct 30 18:37:45.654825 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:45.654800    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:45.655578 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:45.655563    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:45.655631 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:45.655593    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:45.655631 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:45.655604    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:45.710757 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:45.710728    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found"
Oct 30 18:37:45.767308 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:45.767240    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:32:45 +0000 UTC" deadline="2028-06-27 18:55:25.937448909 +0000 UTC"
Oct 30 18:37:45.767308 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:45.767266    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14544h17m40.170185986s"
Oct 30 18:37:45.811078 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:45.811057    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found"
Oct 30 18:37:45.912123 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:45.912094    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found"
Oct 30 18:37:46.012624 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:46.012584    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found"
Oct 30 18:37:46.069689 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:46.069640    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5fcda9dfeca1f860cdbf0f9fc64e83.slice/crio-a50cbce0de3a3d4ff53e77cedbf55d85b0bd76f421acd1dd22b3c3edb2171962 WatchSource:0}: Error finding container a50cbce0de3a3d4ff53e77cedbf55d85b0bd76f421acd1dd22b3c3edb2171962: Status 404 returned error can't find the container with id a50cbce0de3a3d4ff53e77cedbf55d85b0bd76f421acd1dd22b3c3edb2171962
Oct 30 18:37:46.069924 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:46.069903    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e81f6072b8791c05f541935e51a65e.slice/crio-6c4d929837c0848ffd43b672413822b449f17b7e429c0485e97902f24932bb36 WatchSource:0}: Error finding container 6c4d929837c0848ffd43b672413822b449f17b7e429c0485e97902f24932bb36: Status 404 returned error can't find the container with id 6c4d929837c0848ffd43b672413822b449f17b7e429c0485e97902f24932bb36
Oct 30 18:37:46.075692 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.075671    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:37:46.083554 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.083531    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:46.113036 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:46.113005    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found"
Oct 30 18:37:46.213736 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:46.213711    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found"
Oct 30 18:37:46.217397 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.217379    2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:46.290477 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.290447    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:46.353556 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.353478    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:37:46.354593 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.354578    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:37:46.382854 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.382827    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:46.414031 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.414008    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:37:46.467392 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.467367    2575 apiserver.go:52] "Watching apiserver"
Oct 30 18:37:46.490546 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.490528    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 18:37:46.579034 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.579013    2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:37:46.579351 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.579334    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-gsknn","openshift-ovn-kubernetes/ovnkube-node-n5qjv","kube-system/konnectivity-agent-dvpdt","kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-s7gn8","kube-system/global-pull-secret-syncer-bwbj7","openshift-cluster-node-tuning-operator/tuned-f2b5l","openshift-image-registry/node-ca-5lhq5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5","openshift-multus/multus-additional-cni-plugins-lsgdd","openshift-multus/multus-jnnbn","openshift-network-diagnostics/network-check-target-g92j9","openshift-dns/node-resolver-k4r8f"]
Oct 30 18:37:46.581601 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.581584    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.583089 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.583069    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.584322 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.584284    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dvpdt"
Oct 30 18:37:46.585691 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.585675    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:37:46.586795 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.586779    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.587909 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.587893    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:37:46.588007 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:46.587965    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7gn8" podUID="8dccf064-473f-461c-b188-19495b90f62c"
Oct 30 18:37:46.589215 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.589192    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:37:46.590653 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.590633    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:37:46.590754 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:46.590708    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g92j9" podUID="f93cc289-68ea-4c64-9c47-0c16b1cf5f04"
Oct 30 18:37:46.592318 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.592298    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.593772 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.593753    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:37:46.595424 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.595407    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:46.595494 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:46.595459    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwbj7" podUID="d197c973-f7cd-4dff-b621-71fc64897eeb"
Oct 30 18:37:46.596772 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.596757    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.599966 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.599944    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-run-netns\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.600048 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.599983    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-node-log\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.600048 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600011    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-sys-fs\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.600252 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600069    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sknx\" (UniqueName: \"kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx\") pod \"network-check-target-g92j9\" (UID: \"f93cc289-68ea-4c64-9c47-0c16b1cf5f04\") " pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:37:46.600252 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600107    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-var-lib-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.600252 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600136    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.600252 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600186    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.600252 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600223    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-tmp-dir\") pod \"node-resolver-k4r8f\" (UID: \"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3\") " pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:37:46.600493 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600260    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-registration-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.600493 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600301    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysctl-conf\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.600493 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600354    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-sys\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.600493 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600404    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-systemd-units\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.600493 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-slash\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.600784 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600502    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.600784 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600543    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.600784 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600596    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-hosts-file\") pod \"node-resolver-k4r8f\" (UID: \"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3\") " pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:37:46.600784 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600641    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-modprobe-d\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.600784 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600666    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-run\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.600784 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600709    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-cni-bin\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.600784 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600748    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjjll\" (UniqueName: \"kubernetes.io/projected/146371eb-2e54-477f-a7ba-c13450f54828-kube-api-access-xjjll\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.600988 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600789    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce2cd81e-eece-495b-98d8-73fc05aef957-serviceca\") pod \"node-ca-5lhq5\" (UID: \"ce2cd81e-eece-495b-98d8-73fc05aef957\") " pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:37:46.600988 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600819    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pgvf\" (UniqueName: \"kubernetes.io/projected/ce2cd81e-eece-495b-98d8-73fc05aef957-kube-api-access-7pgvf\") pod \"node-ca-5lhq5\" (UID: \"ce2cd81e-eece-495b-98d8-73fc05aef957\") " pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:37:46.600988 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600841    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-kubelet\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.600988 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600864    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a647480-88ca-47a2-9377-2841ce2e14f9-ovn-node-metrics-cert\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.600988 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600934    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-device-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.600988 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600966    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-system-cni-dir\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.601163 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.600991    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.601163 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601029    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b-konnectivity-ca\") pod \"konnectivity-agent-dvpdt\" (UID: \"5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b\") " pod="kube-system/konnectivity-agent-dvpdt"
Oct 30 18:37:46.601163 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601049    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-kubernetes\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.601163 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601063    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/146371eb-2e54-477f-a7ba-c13450f54828-tmp\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.601163 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601083    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nspw\" (UniqueName: \"kubernetes.io/projected/38cac2ec-1eff-4426-a43b-ebe89a226412-kube-api-access-5nspw\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.601163 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-etc-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.601369 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601162    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-cni-netd\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.601369 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601225    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-ovnkube-script-lib\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.601369 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601260    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnb4p\" (UniqueName: \"kubernetes.io/projected/cbf4cda5-bce1-417d-94db-213431b7dd0e-kube-api-access-rnb4p\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.601369 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601310    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78zp5\" (UniqueName: \"kubernetes.io/projected/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-kube-api-access-78zp5\") pod \"node-resolver-k4r8f\" (UID: \"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3\") " pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:37:46.601369 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601340    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-var-lib-kubelet\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.601369 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601359    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-host\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.601581 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601399    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gsgj\" (UniqueName: \"kubernetes.io/projected/8dccf064-473f-461c-b188-19495b90f62c-kube-api-access-9gsgj\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:37:46.601581 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601426    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.601581 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601454    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-ovn\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.601581 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601506    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-log-socket\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.601581 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601541    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.601758 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601596    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-ovnkube-config\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.601758 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601635    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-env-overrides\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.601758 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601681    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7s62\" (UniqueName: \"kubernetes.io/projected/2a647480-88ca-47a2-9377-2841ce2e14f9-kube-api-access-c7s62\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.601758 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601721    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-cni-binary-copy\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.601927 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601766    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b-agent-certs\") pod \"konnectivity-agent-dvpdt\" (UID: \"5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b\") " pod="kube-system/konnectivity-agent-dvpdt"
Oct 30 18:37:46.601927 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601795    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysconfig\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.601927 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601843    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-socket-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.601927 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601882    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.602122 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601937    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysctl-d\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.602122 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.601974    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-lib-modules\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.602122 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.602015    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/146371eb-2e54-477f-a7ba-c13450f54828-etc-tuned\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.602122 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.602059    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:37:46.602122 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.602102    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-cnibin\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.602332 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.602146    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-systemd\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.602332 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.602174    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-systemd\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.602332 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.602226    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce2cd81e-eece-495b-98d8-73fc05aef957-host\") pod \"node-ca-5lhq5\" (UID: \"ce2cd81e-eece-495b-98d8-73fc05aef957\") " pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:37:46.640131 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.639875    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:37:46.640131 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.640088    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6fhn6\""
Oct 30 18:37:46.640131 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.640096    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:37:46.640131 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.640100    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:37:46.642361 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.640996    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:37:46.642361 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.641030    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:37:46.642361 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.641155    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:37:46.642361 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.641221    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x4gd2\""
Oct 30 18:37:46.642361 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.641273    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:37:46.642361 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.641449    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:37:46.642361 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.641963    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:37:46.642361 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.642012    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:37:46.642361 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.642110    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-rrn4s\""
Oct 30 18:37:46.642922 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.642877    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal" event={"ID":"0b5fcda9dfeca1f860cdbf0f9fc64e83","Type":"ContainerStarted","Data":"a50cbce0de3a3d4ff53e77cedbf55d85b0bd76f421acd1dd22b3c3edb2171962"}
Oct 30 18:37:46.643414 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.643395    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:37:46.643574 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.643557    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:37:46.643688 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.643593    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-86nv8\""
Oct 30 18:37:46.643995 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.643978    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal" event={"ID":"80e81f6072b8791c05f541935e51a65e","Type":"ContainerStarted","Data":"6c4d929837c0848ffd43b672413822b449f17b7e429c0485e97902f24932bb36"}
Oct 30 18:37:46.644762 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.644748    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:37:46.644801 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.644766    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:37:46.644801 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.644776    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w46g4\""
Oct 30 18:37:46.646894 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.646878    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:37:46.647067 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.647048    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:37:46.647122 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.647083    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:37:46.647245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.647204    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-27m54\""
Oct 30 18:37:46.692656 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.692636    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:37:46.696292 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.696280    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:37:46.696652 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.696636    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vpq5z\""
Oct 30 18:37:46.696728 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.696676    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5c5lj\""
Oct 30 18:37:46.696728 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.696690    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:37:46.696728 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.696726    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q4mss"
Oct 30 18:37:46.696908 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.696892    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:37:46.696978 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.696897    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:37:46.696978 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.696901    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s6x86\""
Oct 30 18:37:46.697108 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.697085    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:37:46.697214 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.697169    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:37:46.697214 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.697170    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:37:46.697214 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.697183    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:37:46.697214 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.697200    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:37:46.697355 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.697232    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:37:46.697355 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.697324    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:37:46.702884 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.702867    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-sys-fs\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.702936 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.702893    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sknx\" (UniqueName: \"kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx\") pod \"network-check-target-g92j9\" (UID: \"f93cc289-68ea-4c64-9c47-0c16b1cf5f04\") " pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:37:46.702936 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.702911    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-var-lib-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.703007 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.702932    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.703007 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.702959    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.703007 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.702997    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-var-lib-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.703111 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703011    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.703111 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.702998    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-sys-fs\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.703111 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703025    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-tmp-dir\") pod \"node-resolver-k4r8f\" (UID: \"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3\") " pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:37:46.703111 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.702998    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.703111 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703062    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-cni-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.703111 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703086    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-registration-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.703355 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703111    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysctl-conf\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.703355 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703135    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-sys\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.703355 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703120    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-registration-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.703355 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703180    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b6f291a-fb97-4253-b3f1-39d0d482ca88-cni-binary-copy\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.703355 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703207    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-conf-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.703355 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703233    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-systemd-units\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.703355 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703242    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysctl-conf\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.703355 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703265    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-sys\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.703355 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703282    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-systemd-units\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.703355 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703283    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-slash\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.703355 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703308    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-tmp-dir\") pod \"node-resolver-k4r8f\" (UID: \"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3\") " pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:37:46.703355 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703324    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-slash\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.703355 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703355    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.703970 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703387    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.703970 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703414    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-hosts-file\") pod \"node-resolver-k4r8f\" (UID: \"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3\") " pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:37:46.703970 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703441    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-modprobe-d\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.703970 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703456    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.703970 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-run\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.703970 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703504    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-cni-bin\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.703970 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703527    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-cni-bin\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.703970 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703533    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjjll\" (UniqueName: \"kubernetes.io/projected/146371eb-2e54-477f-a7ba-c13450f54828-kube-api-access-xjjll\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.703970 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703558    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce2cd81e-eece-495b-98d8-73fc05aef957-serviceca\") pod \"node-ca-5lhq5\" (UID: \"ce2cd81e-eece-495b-98d8-73fc05aef957\") " pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:37:46.703970 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703573    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgvf\" (UniqueName: \"kubernetes.io/projected/ce2cd81e-eece-495b-98d8-73fc05aef957-kube-api-access-7pgvf\") pod \"node-ca-5lhq5\" (UID: \"ce2cd81e-eece-495b-98d8-73fc05aef957\") " pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:37:46.703970 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703593    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9defd701-4377-47c0-bfaa-a298d0acc866-iptables-alerter-script\") pod \"iptables-alerter-gsknn\" (UID: \"9defd701-4377-47c0-bfaa-a298d0acc866\") " pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:37:46.703970 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703528    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-modprobe-d\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.703970 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703615    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9defd701-4377-47c0-bfaa-a298d0acc866-host-slash\") pod \"iptables-alerter-gsknn\" (UID: \"9defd701-4377-47c0-bfaa-a298d0acc866\") " pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:37:46.703970 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703507    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-run\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.703970 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703656    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d197c973-f7cd-4dff-b621-71fc64897eeb-kubelet-config\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:46.703970 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703772    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d197c973-f7cd-4dff-b621-71fc64897eeb-dbus\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:46.703970 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703802    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-kubelet\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.704813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703840    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a647480-88ca-47a2-9377-2841ce2e14f9-ovn-node-metrics-cert\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.704813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703869    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-device-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.704813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703883    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-kubelet\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.704813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703895    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-system-cni-dir\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.704813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703921    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.704813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703955    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b-konnectivity-ca\") pod \"konnectivity-agent-dvpdt\" (UID: \"5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b\") " pod="kube-system/konnectivity-agent-dvpdt"
Oct 30 18:37:46.704813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.703985    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-kubernetes\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.704813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704002    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.704813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704019    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/146371eb-2e54-477f-a7ba-c13450f54828-tmp\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.704813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704062    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nspw\" (UniqueName: \"kubernetes.io/projected/38cac2ec-1eff-4426-a43b-ebe89a226412-kube-api-access-5nspw\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.704813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704077    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-system-cni-dir\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.704813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704094    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-cnibin\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.704813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704156    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-kubernetes\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.704813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704213    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-k8s-cni-cncf-io\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.704813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704242    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.704813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704253    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-cni-multus\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.704813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704293    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2l6s\" (UniqueName: \"kubernetes.io/projected/5b6f291a-fb97-4253-b3f1-39d0d482ca88-kube-api-access-w2l6s\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704330    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-etc-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704358    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-cni-netd\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704389    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-ovnkube-script-lib\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704417    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnb4p\" (UniqueName: \"kubernetes.io/projected/cbf4cda5-bce1-417d-94db-213431b7dd0e-kube-api-access-rnb4p\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704448    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78zp5\" (UniqueName: \"kubernetes.io/projected/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-kube-api-access-78zp5\") pod \"node-resolver-k4r8f\" (UID: \"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3\") " pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704458    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-cni-netd\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704018    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-hosts-file\") pod \"node-resolver-k4r8f\" (UID: \"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3\") " pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704472    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-var-lib-kubelet\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704461    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-device-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704496    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-host\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704512    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-etc-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704521    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gsgj\" (UniqueName: \"kubernetes.io/projected/8dccf064-473f-461c-b188-19495b90f62c-kube-api-access-9gsgj\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704547    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce2cd81e-eece-495b-98d8-73fc05aef957-serviceca\") pod \"node-ca-5lhq5\" (UID: \"ce2cd81e-eece-495b-98d8-73fc05aef957\") " pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704562    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704591    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-ovn\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704614    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-host\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704619    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-log-socket\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.705419 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704457    2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:37:46.707245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704647    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.707245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704689    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-log-socket\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.707245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704697    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.707245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704700    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-var-lib-kubelet\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.707245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704734    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-ovnkube-config\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.707245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704747    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.707245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704763    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b-konnectivity-ca\") pod \"konnectivity-agent-dvpdt\" (UID: \"5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b\") " pod="kube-system/konnectivity-agent-dvpdt"
Oct 30 18:37:46.707245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704782    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-ovn\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.707245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704798    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-env-overrides\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.707245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704827    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7s62\" (UniqueName: \"kubernetes.io/projected/2a647480-88ca-47a2-9377-2841ce2e14f9-kube-api-access-c7s62\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.707245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.704859    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-kubelet\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.707245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705010    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-cni-binary-copy\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.707245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705054    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b-agent-certs\") pod \"konnectivity-agent-dvpdt\" (UID: \"5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b\") " pod="kube-system/konnectivity-agent-dvpdt"
Oct 30 18:37:46.707245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705082    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysconfig\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.707245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705096    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-ovnkube-script-lib\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.707245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705112    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw99q\" (UniqueName: \"kubernetes.io/projected/9defd701-4377-47c0-bfaa-a298d0acc866-kube-api-access-zw99q\") pod \"iptables-alerter-gsknn\" (UID: \"9defd701-4377-47c0-bfaa-a298d0acc866\") " pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:37:46.707245 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-socket-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705192    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysconfig\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705204    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705254    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysctl-d\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705272    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-env-overrides\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705278    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-lib-modules\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705327    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/146371eb-2e54-477f-a7ba-c13450f54828-etc-tuned\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705355    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705385    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-socket-dir-parent\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705412    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-cnibin\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705429    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysctl-d\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705439    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-systemd\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705356    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-lib-modules\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705494    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-netns\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705525    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-cni-binary-copy\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705538    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-cnibin\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705589    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-systemd\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705590    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-cni-bin\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.708127 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:46.705670    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:46.708935 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705703    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.708935 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705648    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-socket-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.708935 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:46.705745    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs podName:8dccf064-473f-461c-b188-19495b90f62c nodeName:}" failed. No retries permitted until 2026-10-30 18:37:47.205707174 +0000 UTC m=+3.056741071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs") pod "network-metrics-daemon-s7gn8" (UID: "8dccf064-473f-461c-b188-19495b90f62c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:46.708935 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705768    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-systemd\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.708935 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705791    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-systemd\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.708935 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce2cd81e-eece-495b-98d8-73fc05aef957-host\") pod \"node-ca-5lhq5\" (UID: \"ce2cd81e-eece-495b-98d8-73fc05aef957\") " pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:37:46.708935 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705826    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce2cd81e-eece-495b-98d8-73fc05aef957-host\") pod \"node-ca-5lhq5\" (UID: \"ce2cd81e-eece-495b-98d8-73fc05aef957\") " pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:37:46.708935 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705850    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-hostroot\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.708935 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705880    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-daemon-config\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.708935 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705922    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-run-netns\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.708935 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705946    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-etc-kubernetes\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.708935 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705965    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:46.708935 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.705983    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-system-cni-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.708935 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.706008    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-node-log\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.708935 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.706022    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-run-netns\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.708935 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.706049    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-multus-certs\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.708935 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.706132    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-node-log\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.709500 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.706463    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-ovnkube-config\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.709500 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.708412    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/146371eb-2e54-477f-a7ba-c13450f54828-etc-tuned\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.709500 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.708515    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/146371eb-2e54-477f-a7ba-c13450f54828-tmp\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.709500 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.708695    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a647480-88ca-47a2-9377-2841ce2e14f9-ovn-node-metrics-cert\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.709500 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.709108    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b-agent-certs\") pod \"konnectivity-agent-dvpdt\" (UID: \"5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b\") " pod="kube-system/konnectivity-agent-dvpdt"
Oct 30 18:37:46.709500 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.709325    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:46.768245 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:46.768220    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:46.768245 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:46.768246    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:46.768357 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:46.768259    2575 projected.go:196] Error preparing data for projected volume kube-api-access-6sknx for pod openshift-network-diagnostics/network-check-target-g92j9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:46.768357 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:46.768307    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx podName:f93cc289-68ea-4c64-9c47-0c16b1cf5f04 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:47.268293689 +0000 UTC m=+3.119327585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6sknx" (UniqueName: "kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx") pod "network-check-target-g92j9" (UID: "f93cc289-68ea-4c64-9c47-0c16b1cf5f04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:46.772952 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.772928    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78zp5\" (UniqueName: \"kubernetes.io/projected/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-kube-api-access-78zp5\") pod \"node-resolver-k4r8f\" (UID: \"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3\") " pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:37:46.773074 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.772959    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pgvf\" (UniqueName: \"kubernetes.io/projected/ce2cd81e-eece-495b-98d8-73fc05aef957-kube-api-access-7pgvf\") pod \"node-ca-5lhq5\" (UID: \"ce2cd81e-eece-495b-98d8-73fc05aef957\") " pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:37:46.773239 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.773215    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nspw\" (UniqueName: \"kubernetes.io/projected/38cac2ec-1eff-4426-a43b-ebe89a226412-kube-api-access-5nspw\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.773386 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.773366    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjjll\" (UniqueName: \"kubernetes.io/projected/146371eb-2e54-477f-a7ba-c13450f54828-kube-api-access-xjjll\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.773495 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.773477    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnb4p\" (UniqueName: \"kubernetes.io/projected/cbf4cda5-bce1-417d-94db-213431b7dd0e-kube-api-access-rnb4p\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.775528 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.775505    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7s62\" (UniqueName: \"kubernetes.io/projected/2a647480-88ca-47a2-9377-2841ce2e14f9-kube-api-access-c7s62\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.777360 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.777343    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gsgj\" (UniqueName: \"kubernetes.io/projected/8dccf064-473f-461c-b188-19495b90f62c-kube-api-access-9gsgj\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:37:46.806687 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.806666    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:46.806804 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.806701    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-system-cni-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.806804 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.806727    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-multus-certs\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.806804 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.806779    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-multus-certs\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.806963 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.806804    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-system-cni-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.806963 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:46.806819    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:46.806963 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.806858    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-cni-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.806963 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:46.806867    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret podName:d197c973-f7cd-4dff-b621-71fc64897eeb nodeName:}" failed. No retries permitted until 2026-10-30 18:37:47.306854557 +0000 UTC m=+3.157888453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret") pod "global-pull-secret-syncer-bwbj7" (UID: "d197c973-f7cd-4dff-b621-71fc64897eeb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:46.806963 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.806890    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b6f291a-fb97-4253-b3f1-39d0d482ca88-cni-binary-copy\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.806963 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.806905    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-conf-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.806963 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.806928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9defd701-4377-47c0-bfaa-a298d0acc866-iptables-alerter-script\") pod \"iptables-alerter-gsknn\" (UID: \"9defd701-4377-47c0-bfaa-a298d0acc866\") " pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:37:46.806963 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.806935    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-cni-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.806963 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.806945    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9defd701-4377-47c0-bfaa-a298d0acc866-host-slash\") pod \"iptables-alerter-gsknn\" (UID: \"9defd701-4377-47c0-bfaa-a298d0acc866\") " pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.806979    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9defd701-4377-47c0-bfaa-a298d0acc866-host-slash\") pod \"iptables-alerter-gsknn\" (UID: \"9defd701-4377-47c0-bfaa-a298d0acc866\") " pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.806976    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d197c973-f7cd-4dff-b621-71fc64897eeb-kubelet-config\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807011    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d197c973-f7cd-4dff-b621-71fc64897eeb-dbus\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807020    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d197c973-f7cd-4dff-b621-71fc64897eeb-kubelet-config\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807033    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-cnibin\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807028    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-conf-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807049    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-k8s-cni-cncf-io\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807071    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-cni-multus\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807087    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2l6s\" (UniqueName: \"kubernetes.io/projected/5b6f291a-fb97-4253-b3f1-39d0d482ca88-kube-api-access-w2l6s\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807094    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-cnibin\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807108    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-kubelet\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807126    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-cni-multus\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807132    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d197c973-f7cd-4dff-b621-71fc64897eeb-dbus\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807133    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw99q\" (UniqueName: \"kubernetes.io/projected/9defd701-4377-47c0-bfaa-a298d0acc866-kube-api-access-zw99q\") pod \"iptables-alerter-gsknn\" (UID: \"9defd701-4377-47c0-bfaa-a298d0acc866\") " pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807186    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-k8s-cni-cncf-io\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807200    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-kubelet\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807214    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-socket-dir-parent\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.807431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807242    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-netns\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.808061 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807266    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-netns\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.808061 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807267    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-cni-bin\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.808061 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807246    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-socket-dir-parent\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.808061 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807295    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-hostroot\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.808061 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807320    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-daemon-config\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.808061 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807321    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-cni-bin\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.808061 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807344    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-etc-kubernetes\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.808061 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807386    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-hostroot\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.808061 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.807404    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-etc-kubernetes\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.808340 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.808095    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-daemon-config\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.808340 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.808194    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9defd701-4377-47c0-bfaa-a298d0acc866-iptables-alerter-script\") pod \"iptables-alerter-gsknn\" (UID: \"9defd701-4377-47c0-bfaa-a298d0acc866\") " pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:37:46.808340 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.808196    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b6f291a-fb97-4253-b3f1-39d0d482ca88-cni-binary-copy\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.874285 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.874226    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw99q\" (UniqueName: \"kubernetes.io/projected/9defd701-4377-47c0-bfaa-a298d0acc866-kube-api-access-zw99q\") pod \"iptables-alerter-gsknn\" (UID: \"9defd701-4377-47c0-bfaa-a298d0acc866\") " pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:37:46.874390 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.874338    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2l6s\" (UniqueName: \"kubernetes.io/projected/5b6f291a-fb97-4253-b3f1-39d0d482ca88-kube-api-access-w2l6s\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.895205 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.895189    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:37:46.904384 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:46.904361    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf4cda5_bce1_417d_94db_213431b7dd0e.slice/crio-d74739a98b7763906ca579fafc2acb7039362b0f99d932c6a7c22e376ecc1f10 WatchSource:0}: Error finding container d74739a98b7763906ca579fafc2acb7039362b0f99d932c6a7c22e376ecc1f10: Status 404 returned error can't find the container with id d74739a98b7763906ca579fafc2acb7039362b0f99d932c6a7c22e376ecc1f10
Oct 30 18:37:46.904978 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.904953    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:37:46.908024 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.908004    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dvpdt"
Oct 30 18:37:46.912407 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.912390    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:37:46.912699 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:46.912677    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38cac2ec_1eff_4426_a43b_ebe89a226412.slice/crio-429f48f2a4437f4a2921d6230babddf6639f995add43b4c25f613eda43984e89 WatchSource:0}: Error finding container 429f48f2a4437f4a2921d6230babddf6639f995add43b4c25f613eda43984e89: Status 404 returned error can't find the container with id 429f48f2a4437f4a2921d6230babddf6639f995add43b4c25f613eda43984e89
Oct 30 18:37:46.916400 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:46.916380    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff8c95a_a5f2_41d1_91f3_ff288ebf2e5b.slice/crio-9ea384eecbbf339ff3da088dbeb1dcd0cd4559b14e8176236caa9ce344a36a01 WatchSource:0}: Error finding container 9ea384eecbbf339ff3da088dbeb1dcd0cd4559b14e8176236caa9ce344a36a01: Status 404 returned error can't find the container with id 9ea384eecbbf339ff3da088dbeb1dcd0cd4559b14e8176236caa9ce344a36a01
Oct 30 18:37:46.917988 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.917967    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:37:46.921323 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:46.921303    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b3ccb2_2cc4_4291_a39f_e2edfff638f3.slice/crio-baa3f0a35e2d65e89e4bc0f526c85bf91ba4af35a01193febb924f5e9f91f644 WatchSource:0}: Error finding container baa3f0a35e2d65e89e4bc0f526c85bf91ba4af35a01193febb924f5e9f91f644: Status 404 returned error can't find the container with id baa3f0a35e2d65e89e4bc0f526c85bf91ba4af35a01193febb924f5e9f91f644
Oct 30 18:37:46.924649 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.924627    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:37:46.925926 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:46.925908    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146371eb_2e54_477f_a7ba_c13450f54828.slice/crio-d7eade0fb0c3b13dae65501b336eab1a47c26cabbe59ca3f6cec2ca5181cc574 WatchSource:0}: Error finding container d7eade0fb0c3b13dae65501b336eab1a47c26cabbe59ca3f6cec2ca5181cc574: Status 404 returned error can't find the container with id d7eade0fb0c3b13dae65501b336eab1a47c26cabbe59ca3f6cec2ca5181cc574
Oct 30 18:37:46.932488 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:46.932468    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2cd81e_eece_495b_98d8_73fc05aef957.slice/crio-2325f2ad51e14b7934614b25dc3c88bf2bf6a6027136849332fa220e958903fc WatchSource:0}: Error finding container 2325f2ad51e14b7934614b25dc3c88bf2bf6a6027136849332fa220e958903fc: Status 404 returned error can't find the container with id 2325f2ad51e14b7934614b25dc3c88bf2bf6a6027136849332fa220e958903fc
Oct 30 18:37:46.937656 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.937629    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:37:46.939530 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.939511    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:37:46.942223 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:46.942188    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jnnbn"
Oct 30 18:37:46.945684 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:46.945497    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a647480_88ca_47a2_9377_2841ce2e14f9.slice/crio-a6b8df729ef95c302ae3ad908c1a598848f2daa57e6b3f16555403a1e390805a WatchSource:0}: Error finding container a6b8df729ef95c302ae3ad908c1a598848f2daa57e6b3f16555403a1e390805a: Status 404 returned error can't find the container with id a6b8df729ef95c302ae3ad908c1a598848f2daa57e6b3f16555403a1e390805a
Oct 30 18:37:46.948971 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:46.948868    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9defd701_4377_47c0_bfaa_a298d0acc866.slice/crio-f03e3791eff44f8f48ae35a5363d1ce851f7733d1d9c4df0c76b8e934e85aaf7 WatchSource:0}: Error finding container f03e3791eff44f8f48ae35a5363d1ce851f7733d1d9c4df0c76b8e934e85aaf7: Status 404 returned error can't find the container with id f03e3791eff44f8f48ae35a5363d1ce851f7733d1d9c4df0c76b8e934e85aaf7
Oct 30 18:37:46.953774 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:37:46.953750    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6f291a_fb97_4253_b3f1_39d0d482ca88.slice/crio-27d3510732809f2f8ce65fec367e12425b2394c2fe693a5b390ebacf52247b13 WatchSource:0}: Error finding container 27d3510732809f2f8ce65fec367e12425b2394c2fe693a5b390ebacf52247b13: Status 404 returned error can't find the container with id 27d3510732809f2f8ce65fec367e12425b2394c2fe693a5b390ebacf52247b13
Oct 30 18:37:47.210668 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:47.210574    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:37:47.210847 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:47.210731    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:47.210847 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:47.210805    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs podName:8dccf064-473f-461c-b188-19495b90f62c nodeName:}" failed. No retries permitted until 2026-10-30 18:37:48.210783392 +0000 UTC m=+4.061817286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs") pod "network-metrics-daemon-s7gn8" (UID: "8dccf064-473f-461c-b188-19495b90f62c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:47.311325 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:47.311295    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sknx\" (UniqueName: \"kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx\") pod \"network-check-target-g92j9\" (UID: \"f93cc289-68ea-4c64-9c47-0c16b1cf5f04\") " pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:37:47.311520 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:47.311390    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:47.311520 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:47.311487    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:47.311645 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:47.311568    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret podName:d197c973-f7cd-4dff-b621-71fc64897eeb nodeName:}" failed. No retries permitted until 2026-10-30 18:37:48.311549388 +0000 UTC m=+4.162583279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret") pod "global-pull-secret-syncer-bwbj7" (UID: "d197c973-f7cd-4dff-b621-71fc64897eeb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:47.311713 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:47.311694    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:47.311713 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:47.311712    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:47.311808 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:47.311724    2575 projected.go:196] Error preparing data for projected volume kube-api-access-6sknx for pod openshift-network-diagnostics/network-check-target-g92j9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:47.311808 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:47.311764    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx podName:f93cc289-68ea-4c64-9c47-0c16b1cf5f04 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:48.31175135 +0000 UTC m=+4.162785228 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6sknx" (UniqueName: "kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx") pod "network-check-target-g92j9" (UID: "f93cc289-68ea-4c64-9c47-0c16b1cf5f04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:47.647962 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:47.647921    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gsknn" event={"ID":"9defd701-4377-47c0-bfaa-a298d0acc866","Type":"ContainerStarted","Data":"f03e3791eff44f8f48ae35a5363d1ce851f7733d1d9c4df0c76b8e934e85aaf7"}
Oct 30 18:37:47.649743 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:47.649713    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv" event={"ID":"2a647480-88ca-47a2-9377-2841ce2e14f9","Type":"ContainerStarted","Data":"a6b8df729ef95c302ae3ad908c1a598848f2daa57e6b3f16555403a1e390805a"}
Oct 30 18:37:47.650975 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:47.650947    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f2b5l" event={"ID":"146371eb-2e54-477f-a7ba-c13450f54828","Type":"ContainerStarted","Data":"d7eade0fb0c3b13dae65501b336eab1a47c26cabbe59ca3f6cec2ca5181cc574"}
Oct 30 18:37:47.652403 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:47.652374    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dvpdt" event={"ID":"5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b","Type":"ContainerStarted","Data":"9ea384eecbbf339ff3da088dbeb1dcd0cd4559b14e8176236caa9ce344a36a01"}
Oct 30 18:37:47.653800 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:47.653777    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsgdd" event={"ID":"38cac2ec-1eff-4426-a43b-ebe89a226412","Type":"ContainerStarted","Data":"429f48f2a4437f4a2921d6230babddf6639f995add43b4c25f613eda43984e89"}
Oct 30 18:37:47.655252 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:47.655227    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5" event={"ID":"cbf4cda5-bce1-417d-94db-213431b7dd0e","Type":"ContainerStarted","Data":"d74739a98b7763906ca579fafc2acb7039362b0f99d932c6a7c22e376ecc1f10"}
Oct 30 18:37:47.656742 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:47.656720    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jnnbn" event={"ID":"5b6f291a-fb97-4253-b3f1-39d0d482ca88","Type":"ContainerStarted","Data":"27d3510732809f2f8ce65fec367e12425b2394c2fe693a5b390ebacf52247b13"}
Oct 30 18:37:47.658200 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:47.658128    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5lhq5" event={"ID":"ce2cd81e-eece-495b-98d8-73fc05aef957","Type":"ContainerStarted","Data":"2325f2ad51e14b7934614b25dc3c88bf2bf6a6027136849332fa220e958903fc"}
Oct 30 18:37:47.659968 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:47.659934    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k4r8f" event={"ID":"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3","Type":"ContainerStarted","Data":"baa3f0a35e2d65e89e4bc0f526c85bf91ba4af35a01193febb924f5e9f91f644"}
Oct 30 18:37:47.699251 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:47.699211    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:32:47 +0000 UTC" deadline="2028-07-10 13:29:55.40515838 +0000 UTC"
Oct 30 18:37:47.699384 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:47.699253    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14850h52m7.705909374s"
Oct 30 18:37:48.221669 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:48.221626    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:37:48.221889 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:48.221804    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:48.221889 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:48.221870    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs podName:8dccf064-473f-461c-b188-19495b90f62c nodeName:}" failed. No retries permitted until 2026-10-30 18:37:50.22185153 +0000 UTC m=+6.072885422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs") pod "network-metrics-daemon-s7gn8" (UID: "8dccf064-473f-461c-b188-19495b90f62c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:48.323019 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:48.322944    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:48.323019 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:48.322995    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sknx\" (UniqueName: \"kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx\") pod \"network-check-target-g92j9\" (UID: \"f93cc289-68ea-4c64-9c47-0c16b1cf5f04\") " pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:37:48.323266 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:48.323145    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:48.323266 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:48.323180    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:48.323266 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:48.323194    2575 projected.go:196] Error preparing data for projected volume kube-api-access-6sknx for pod openshift-network-diagnostics/network-check-target-g92j9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:48.323266 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:48.323256    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx podName:f93cc289-68ea-4c64-9c47-0c16b1cf5f04 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:50.323235623 +0000 UTC m=+6.174269521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6sknx" (UniqueName: "kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx") pod "network-check-target-g92j9" (UID: "f93cc289-68ea-4c64-9c47-0c16b1cf5f04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:48.323488 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:48.323340    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:48.323488 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:48.323379    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret podName:d197c973-f7cd-4dff-b621-71fc64897eeb nodeName:}" failed. No retries permitted until 2026-10-30 18:37:50.32336629 +0000 UTC m=+6.174400183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret") pod "global-pull-secret-syncer-bwbj7" (UID: "d197c973-f7cd-4dff-b621-71fc64897eeb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:48.639143 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:48.639114    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:48.639307 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:48.639290    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:37:48.639433 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:48.639405    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7gn8" podUID="8dccf064-473f-461c-b188-19495b90f62c"
Oct 30 18:37:48.639517 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:48.639492    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:37:48.639575 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:48.639546    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g92j9" podUID="f93cc289-68ea-4c64-9c47-0c16b1cf5f04"
Oct 30 18:37:48.641998 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:48.641961    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwbj7" podUID="d197c973-f7cd-4dff-b621-71fc64897eeb"
Oct 30 18:37:48.699573 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:48.699516    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:32:47 +0000 UTC" deadline="2028-05-22 00:31:14.154391791 +0000 UTC"
Oct 30 18:37:48.699573 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:48.699569    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13661h53m25.454824919s"
Oct 30 18:37:49.686325 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:49.686286    2575 generic.go:356] "Generic (PLEG): container finished" podID="0b5fcda9dfeca1f860cdbf0f9fc64e83" containerID="8addf35385b5858e334e824bb2105c5237f40b2b4fdbb755523f62095a88a827" exitCode=0
Oct 30 18:37:49.686506 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:49.686399    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal" event={"ID":"0b5fcda9dfeca1f860cdbf0f9fc64e83","Type":"ContainerDied","Data":"8addf35385b5858e334e824bb2105c5237f40b2b4fdbb755523f62095a88a827"}
Oct 30 18:37:49.693098 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:49.693067    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal" event={"ID":"80e81f6072b8791c05f541935e51a65e","Type":"ContainerStarted","Data":"772dfee152cbb1e473cf868308c663f022b4f35b695c2ef9c13d2d5ca341fd65"}
Oct 30 18:37:50.247251 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:50.247209    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:37:50.247821 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:50.247339    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:50.247821 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:50.247420    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs podName:8dccf064-473f-461c-b188-19495b90f62c nodeName:}" failed. No retries permitted until 2026-10-30 18:37:54.247398651 +0000 UTC m=+10.098432546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs") pod "network-metrics-daemon-s7gn8" (UID: "8dccf064-473f-461c-b188-19495b90f62c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:50.348711 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:50.347985    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sknx\" (UniqueName: \"kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx\") pod \"network-check-target-g92j9\" (UID: \"f93cc289-68ea-4c64-9c47-0c16b1cf5f04\") " pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:37:50.348711 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:50.348187    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:50.348711 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:50.348340    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:50.348711 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:50.348462    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret podName:d197c973-f7cd-4dff-b621-71fc64897eeb nodeName:}" failed. No retries permitted until 2026-10-30 18:37:54.348440939 +0000 UTC m=+10.199474830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret") pod "global-pull-secret-syncer-bwbj7" (UID: "d197c973-f7cd-4dff-b621-71fc64897eeb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:50.348711 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:50.348593    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:50.348711 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:50.348613    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:50.348711 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:50.348635    2575 projected.go:196] Error preparing data for projected volume kube-api-access-6sknx for pod openshift-network-diagnostics/network-check-target-g92j9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:50.348711 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:50.348678    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx podName:f93cc289-68ea-4c64-9c47-0c16b1cf5f04 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:54.348664473 +0000 UTC m=+10.199698368 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6sknx" (UniqueName: "kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx") pod "network-check-target-g92j9" (UID: "f93cc289-68ea-4c64-9c47-0c16b1cf5f04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:50.640887 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:50.640355    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:37:50.640887 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:50.640569    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g92j9" podUID="f93cc289-68ea-4c64-9c47-0c16b1cf5f04"
Oct 30 18:37:50.640887 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:50.640649    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:50.640887 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:50.640756    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwbj7" podUID="d197c973-f7cd-4dff-b621-71fc64897eeb"
Oct 30 18:37:50.640887 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:50.640853    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:37:50.641253 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:50.640949    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7gn8" podUID="8dccf064-473f-461c-b188-19495b90f62c"
Oct 30 18:37:50.705440 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:50.705400    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsgdd" event={"ID":"38cac2ec-1eff-4426-a43b-ebe89a226412","Type":"ContainerStarted","Data":"78b3c6f252b6ae372501530983fb7235880d2e4a319ae550d861dc34d30c2d00"}
Oct 30 18:37:50.718732 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:50.718680    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal" event={"ID":"0b5fcda9dfeca1f860cdbf0f9fc64e83","Type":"ContainerStarted","Data":"12e2dac573cbbb37e34628cfb3db2e4f22d4b87706280f9f1c747f3176097c1c"}
Oct 30 18:37:50.835326 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:50.835252    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal" podStartSLOduration=3.835231378 podStartE2EDuration="3.835231378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:49.940098607 +0000 UTC m=+5.791132506" watchObservedRunningTime="2026-10-30 18:37:50.835231378 +0000 UTC m=+6.686265279"
Oct 30 18:37:51.723996 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:51.723967    2575 generic.go:356] "Generic (PLEG): container finished" podID="38cac2ec-1eff-4426-a43b-ebe89a226412" containerID="78b3c6f252b6ae372501530983fb7235880d2e4a319ae550d861dc34d30c2d00" exitCode=0
Oct 30 18:37:51.724431 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:51.724076    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsgdd" event={"ID":"38cac2ec-1eff-4426-a43b-ebe89a226412","Type":"ContainerDied","Data":"78b3c6f252b6ae372501530983fb7235880d2e4a319ae550d861dc34d30c2d00"}
Oct 30 18:37:51.853772 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:51.853715    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal" podStartSLOduration=4.853695015 podStartE2EDuration="4.853695015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:50.96003911 +0000 UTC m=+6.811073011" watchObservedRunningTime="2026-10-30 18:37:51.853695015 +0000 UTC m=+7.704728915"
Oct 30 18:37:52.641824 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:52.641333    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:37:52.641824 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:52.641479    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7gn8" podUID="8dccf064-473f-461c-b188-19495b90f62c"
Oct 30 18:37:52.641824 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:52.641530    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:37:52.641824 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:52.641577    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g92j9" podUID="f93cc289-68ea-4c64-9c47-0c16b1cf5f04"
Oct 30 18:37:52.641824 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:52.641605    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:52.641824 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:52.641650    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwbj7" podUID="d197c973-f7cd-4dff-b621-71fc64897eeb"
Oct 30 18:37:52.739941 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:52.739907    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dvpdt" event={"ID":"5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b","Type":"ContainerStarted","Data":"54d83cbb0c5b54b8cf77c3f75959c9c0c04c835f2e21cba613ef76d7d9268091"}
Oct 30 18:37:53.666717 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:53.666511    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dvpdt"
Oct 30 18:37:53.666880 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:53.666838    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dvpdt"
Oct 30 18:37:53.748073 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:53.747440    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5lhq5" event={"ID":"ce2cd81e-eece-495b-98d8-73fc05aef957","Type":"ContainerStarted","Data":"df896394af2b416763e466096153d481d7902b30f711eb26c74909e8143f4619"}
Oct 30 18:37:53.748073 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:53.747516    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dvpdt"
Oct 30 18:37:53.748611 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:53.748103    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dvpdt"
Oct 30 18:37:53.794353 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:53.793078    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dvpdt" podStartSLOduration=4.009777601 podStartE2EDuration="8.793057525s" totalImagesPullingTime="4.783279924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:52.869367273 +0000 UTC m=+8.720401173" watchObservedRunningTime="2026-10-30 18:37:53.793057525 +0000 UTC m=+9.644091426"
Oct 30 18:37:54.045748 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:54.045641    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5lhq5" podStartSLOduration=3.370261168 podStartE2EDuration="9.045622656s" totalImagesPullingTime="5.675361488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:54.045038934 +0000 UTC m=+9.896072834" watchObservedRunningTime="2026-10-30 18:37:54.045622656 +0000 UTC m=+9.896656563"
Oct 30 18:37:54.282033 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:54.281455    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:37:54.282033 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:54.281607    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:54.282033 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:54.281673    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs podName:8dccf064-473f-461c-b188-19495b90f62c nodeName:}" failed. No retries permitted until 2026-10-30 18:38:02.281654507 +0000 UTC m=+18.132688400 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs") pod "network-metrics-daemon-s7gn8" (UID: "8dccf064-473f-461c-b188-19495b90f62c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:54.382878 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:54.381861    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:54.382878 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:54.381918    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sknx\" (UniqueName: \"kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx\") pod \"network-check-target-g92j9\" (UID: \"f93cc289-68ea-4c64-9c47-0c16b1cf5f04\") " pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:37:54.382878 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:54.382084    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:54.382878 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:54.382105    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:54.382878 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:54.382122    2575 projected.go:196] Error preparing data for projected volume kube-api-access-6sknx for pod openshift-network-diagnostics/network-check-target-g92j9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:54.382878 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:54.382203    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx podName:f93cc289-68ea-4c64-9c47-0c16b1cf5f04 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:02.382163009 +0000 UTC m=+18.233196900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6sknx" (UniqueName: "kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx") pod "network-check-target-g92j9" (UID: "f93cc289-68ea-4c64-9c47-0c16b1cf5f04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:54.382878 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:54.382629    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:54.382878 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:54.382685    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret podName:d197c973-f7cd-4dff-b621-71fc64897eeb nodeName:}" failed. No retries permitted until 2026-10-30 18:38:02.382667934 +0000 UTC m=+18.233701825 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret") pod "global-pull-secret-syncer-bwbj7" (UID: "d197c973-f7cd-4dff-b621-71fc64897eeb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:54.641937 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:54.640323    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:54.641937 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:54.640452    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwbj7" podUID="d197c973-f7cd-4dff-b621-71fc64897eeb"
Oct 30 18:37:54.641937 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:54.640847    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:37:54.641937 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:54.640961    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7gn8" podUID="8dccf064-473f-461c-b188-19495b90f62c"
Oct 30 18:37:54.641937 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:54.641245    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:37:54.641937 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:54.641334    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g92j9" podUID="f93cc289-68ea-4c64-9c47-0c16b1cf5f04"
Oct 30 18:37:54.754183 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:54.753824    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5" event={"ID":"cbf4cda5-bce1-417d-94db-213431b7dd0e","Type":"ContainerStarted","Data":"360625bcd45f3963805cc858bd62c33bff034c09bcbffa95046e934775d04077"}
Oct 30 18:37:55.759856 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:55.759807    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jnnbn" event={"ID":"5b6f291a-fb97-4253-b3f1-39d0d482ca88","Type":"ContainerStarted","Data":"c60c3dee6b2b40ac24d7353af1c6ad73b1a7aee824b608d64cedd865642a8f1e"}
Oct 30 18:37:55.881592 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:55.881561    2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:37:56.571441 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:56.571324    2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:37:55.88158634Z","UUID":"f82b42b6-f2b4-40d9-88ba-0312da0213b6","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:37:56.573048 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:56.573025    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:37:56.573048 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:56.573059    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:37:56.638791 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:56.638329    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:37:56.638791 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:56.638380    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:37:56.638791 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:56.638413    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:56.638791 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:56.638483    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g92j9" podUID="f93cc289-68ea-4c64-9c47-0c16b1cf5f04"
Oct 30 18:37:56.638791 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:56.638625    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7gn8" podUID="8dccf064-473f-461c-b188-19495b90f62c"
Oct 30 18:37:56.638791 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:56.638712    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwbj7" podUID="d197c973-f7cd-4dff-b621-71fc64897eeb"
Oct 30 18:37:56.765583 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:56.765530    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsgdd" event={"ID":"38cac2ec-1eff-4426-a43b-ebe89a226412","Type":"ContainerStarted","Data":"2498eeaeeca6982e97df4e9516a2e32acca3e6b6d708d75e4bd25aaca807ab97"}
Oct 30 18:37:56.774055 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:56.774020    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5" event={"ID":"cbf4cda5-bce1-417d-94db-213431b7dd0e","Type":"ContainerStarted","Data":"74d8099921a14caa256e2d260b7a8a3485092f8ba362d13b6c7d72afef741152"}
Oct 30 18:37:56.775814 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:56.775747    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k4r8f" event={"ID":"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3","Type":"ContainerStarted","Data":"1ca4c9c5a1d7e28fcac7c1918867c556e463947ac2c05ee5399a735b252d6f1a"}
Oct 30 18:37:56.890656 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:56.890583    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jnnbn" podStartSLOduration=3.259702346 podStartE2EDuration="11.890563946s" totalImagesPullingTime="8.6308616s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:55.891401044 +0000 UTC m=+11.742434944" watchObservedRunningTime="2026-10-30 18:37:56.890563946 +0000 UTC m=+12.741597842"
Oct 30 18:37:57.780409 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:57.780372    2575 generic.go:356] "Generic (PLEG): container finished" podID="38cac2ec-1eff-4426-a43b-ebe89a226412" containerID="2498eeaeeca6982e97df4e9516a2e32acca3e6b6d708d75e4bd25aaca807ab97" exitCode=0
Oct 30 18:37:57.780997 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:57.780472    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsgdd" event={"ID":"38cac2ec-1eff-4426-a43b-ebe89a226412","Type":"ContainerDied","Data":"2498eeaeeca6982e97df4e9516a2e32acca3e6b6d708d75e4bd25aaca807ab97"}
Oct 30 18:37:57.783242 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:57.783209    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gsknn" event={"ID":"9defd701-4377-47c0-bfaa-a298d0acc866","Type":"ContainerStarted","Data":"0cc5520af7b15c4c7151082117a79b76443712d6603611abf711afd5dee20713"}
Oct 30 18:37:57.952797 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:57.952759    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-k4r8f" podStartSLOduration=3.655141322 podStartE2EDuration="12.952746887s" totalImagesPullingTime="9.297605565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:57.006372206 +0000 UTC m=+12.857406104" watchObservedRunningTime="2026-10-30 18:37:57.952746887 +0000 UTC m=+13.803780782"
Oct 30 18:37:58.067580 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:58.067537    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gsknn" podStartSLOduration=4.055867627 podStartE2EDuration="13.067524429s" totalImagesPullingTime="9.011656802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:58.067019778 +0000 UTC m=+13.918053675" watchObservedRunningTime="2026-10-30 18:37:58.067524429 +0000 UTC m=+13.918558325"
Oct 30 18:37:58.639249 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:58.638928    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:37:58.639249 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:58.639042    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:37:58.639435 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:58.639283    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g92j9" podUID="f93cc289-68ea-4c64-9c47-0c16b1cf5f04"
Oct 30 18:37:58.639435 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:58.639377    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwbj7" podUID="d197c973-f7cd-4dff-b621-71fc64897eeb"
Oct 30 18:37:58.639435 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:58.639053    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:37:58.639549 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:37:58.639477    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7gn8" podUID="8dccf064-473f-461c-b188-19495b90f62c"
Oct 30 18:37:58.787502 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:58.787467    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f2b5l" event={"ID":"146371eb-2e54-477f-a7ba-c13450f54828","Type":"ContainerStarted","Data":"51e621a391a826b01af54664fef42c041f8e0229ca0aff6ec0f82240ad3bcc31"}
Oct 30 18:37:58.789956 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:58.789927    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5" event={"ID":"cbf4cda5-bce1-417d-94db-213431b7dd0e","Type":"ContainerStarted","Data":"287dacf9329582bd9d6cdf7e493211ec31b99b742066418c3b5f521a471c7f02"}
Oct 30 18:37:58.904473 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:58.904365    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f2b5l" podStartSLOduration=2.513037872 podStartE2EDuration="13.904347753s" totalImagesPullingTime="11.391309881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:58.903639865 +0000 UTC m=+14.754673761" watchObservedRunningTime="2026-10-30 18:37:58.904347753 +0000 UTC m=+14.755381651"
Oct 30 18:37:59.027753 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:59.027699    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5" podStartSLOduration=3.317332601 podStartE2EDuration="14.027681664s" totalImagesPullingTime="10.710349063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:59.027322605 +0000 UTC m=+14.878356502" watchObservedRunningTime="2026-10-30 18:37:59.027681664 +0000 UTC m=+14.878715563"
Oct 30 18:37:59.794726 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:59.794691    2575 generic.go:356] "Generic (PLEG): container finished" podID="38cac2ec-1eff-4426-a43b-ebe89a226412" containerID="65e19c8151db95d6589f1f4c74b3a0a99da838bf1772596388d185bfbda6990c" exitCode=0
Oct 30 18:37:59.795172 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:37:59.794779    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsgdd" event={"ID":"38cac2ec-1eff-4426-a43b-ebe89a226412","Type":"ContainerDied","Data":"65e19c8151db95d6589f1f4c74b3a0a99da838bf1772596388d185bfbda6990c"}
Oct 30 18:38:00.638698 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:00.638661    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:00.638698 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:00.638667    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:00.638934 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:00.638819    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7gn8" podUID="8dccf064-473f-461c-b188-19495b90f62c"
Oct 30 18:38:00.638934 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:00.638902    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:00.639033 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:00.638986    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwbj7" podUID="d197c973-f7cd-4dff-b621-71fc64897eeb"
Oct 30 18:38:00.639163 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:00.639040    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g92j9" podUID="f93cc289-68ea-4c64-9c47-0c16b1cf5f04"
Oct 30 18:38:01.801719 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:01.801674    2575 generic.go:356] "Generic (PLEG): container finished" podID="38cac2ec-1eff-4426-a43b-ebe89a226412" containerID="7b159e8ee3084f0bff30ad8faa7f3688b3a885716fd4904b4c00235f25496b64" exitCode=0
Oct 30 18:38:01.802206 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:01.801730    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsgdd" event={"ID":"38cac2ec-1eff-4426-a43b-ebe89a226412","Type":"ContainerDied","Data":"7b159e8ee3084f0bff30ad8faa7f3688b3a885716fd4904b4c00235f25496b64"}
Oct 30 18:38:02.347905 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:02.347882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:02.348000 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:02.347987    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:02.348051 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:02.348043    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs podName:8dccf064-473f-461c-b188-19495b90f62c nodeName:}" failed. No retries permitted until 2026-10-30 18:38:18.34803112 +0000 UTC m=+34.199065007 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs") pod "network-metrics-daemon-s7gn8" (UID: "8dccf064-473f-461c-b188-19495b90f62c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:02.448499 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:02.448474    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:02.448620 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:02.448510    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sknx\" (UniqueName: \"kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx\") pod \"network-check-target-g92j9\" (UID: \"f93cc289-68ea-4c64-9c47-0c16b1cf5f04\") " pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:02.448686 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:02.448628    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:02.448686 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:02.448640    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:38:02.448686 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:02.448656    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:38:02.448686 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:02.448668    2575 projected.go:196] Error preparing data for projected volume kube-api-access-6sknx for pod openshift-network-diagnostics/network-check-target-g92j9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:02.448880 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:02.448691    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret podName:d197c973-f7cd-4dff-b621-71fc64897eeb nodeName:}" failed. No retries permitted until 2026-10-30 18:38:18.448673837 +0000 UTC m=+34.299707731 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret") pod "global-pull-secret-syncer-bwbj7" (UID: "d197c973-f7cd-4dff-b621-71fc64897eeb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:02.448880 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:02.448708    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx podName:f93cc289-68ea-4c64-9c47-0c16b1cf5f04 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:18.44869972 +0000 UTC m=+34.299733594 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6sknx" (UniqueName: "kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx") pod "network-check-target-g92j9" (UID: "f93cc289-68ea-4c64-9c47-0c16b1cf5f04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:02.638275 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:02.638251    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:02.638275 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:02.638265    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:02.638429 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:02.638275    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:02.638429 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:02.638362    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g92j9" podUID="f93cc289-68ea-4c64-9c47-0c16b1cf5f04"
Oct 30 18:38:02.638584 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:02.638428    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwbj7" podUID="d197c973-f7cd-4dff-b621-71fc64897eeb"
Oct 30 18:38:02.638584 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:02.638526    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7gn8" podUID="8dccf064-473f-461c-b188-19495b90f62c"
Oct 30 18:38:02.807265 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:02.807235    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 18:38:02.810810 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:02.810718    2575 generic.go:356] "Generic (PLEG): container finished" podID="2a647480-88ca-47a2-9377-2841ce2e14f9" containerID="edf1c2068373cb6c246d9093fc6f281031d75fd96acea08bea6c228d0e099909" exitCode=1
Oct 30 18:38:02.810810 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:02.810765    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv" event={"ID":"2a647480-88ca-47a2-9377-2841ce2e14f9","Type":"ContainerStarted","Data":"658ec960a05e1e57d433af346b8ac811ccf6726da6d35aa372ced71f55a850e1"}
Oct 30 18:38:02.810976 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:02.810885    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv" event={"ID":"2a647480-88ca-47a2-9377-2841ce2e14f9","Type":"ContainerStarted","Data":"ac8294dfe32e7ebfc4aaaff8786e8c79fabd505398bd340c4ff138d0405baf5f"}
Oct 30 18:38:02.810976 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:02.810931    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv" event={"ID":"2a647480-88ca-47a2-9377-2841ce2e14f9","Type":"ContainerStarted","Data":"5d539b817c9965ebbb358dd9d41e377b0c2b6e975a2bc56744dee2c07975563c"}
Oct 30 18:38:02.810976 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:02.810975    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv" event={"ID":"2a647480-88ca-47a2-9377-2841ce2e14f9","Type":"ContainerStarted","Data":"a27711f984ed0956c0f29f0e954a9841f43b03a0e5c399b313412a6667baf792"}
Oct 30 18:38:02.811096 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:02.811028    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv" event={"ID":"2a647480-88ca-47a2-9377-2841ce2e14f9","Type":"ContainerDied","Data":"edf1c2068373cb6c246d9093fc6f281031d75fd96acea08bea6c228d0e099909"}
Oct 30 18:38:02.811096 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:02.811073    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv" event={"ID":"2a647480-88ca-47a2-9377-2841ce2e14f9","Type":"ContainerStarted","Data":"5de12ba9911226b35e5b53c1aa6872f321bf2d5e866792e314440e99299fde63"}
Oct 30 18:38:03.815458 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:03.815427    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsgdd" event={"ID":"38cac2ec-1eff-4426-a43b-ebe89a226412","Type":"ContainerStarted","Data":"e787d8530b3567a8411d49571c3205e250ca554602f1525315156b82c2190629"}
Oct 30 18:38:04.638953 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:04.638920    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:04.639157 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:04.639043    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:04.639157 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:04.639050    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7gn8" podUID="8dccf064-473f-461c-b188-19495b90f62c"
Oct 30 18:38:04.639157 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:04.639119    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwbj7" podUID="d197c973-f7cd-4dff-b621-71fc64897eeb"
Oct 30 18:38:04.639386 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:04.639156    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:04.639386 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:04.639255    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g92j9" podUID="f93cc289-68ea-4c64-9c47-0c16b1cf5f04"
Oct 30 18:38:04.820784 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:04.820758    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 18:38:04.821363 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:04.821091    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv" event={"ID":"2a647480-88ca-47a2-9377-2841ce2e14f9","Type":"ContainerStarted","Data":"e88cbe34b969d3e99dbdb19f0721f90a9bdfccb0b46a8983c59809f05fa0605d"}
Oct 30 18:38:04.823592 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:04.823568    2575 generic.go:356] "Generic (PLEG): container finished" podID="38cac2ec-1eff-4426-a43b-ebe89a226412" containerID="e787d8530b3567a8411d49571c3205e250ca554602f1525315156b82c2190629" exitCode=0
Oct 30 18:38:04.823718 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:04.823608    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsgdd" event={"ID":"38cac2ec-1eff-4426-a43b-ebe89a226412","Type":"ContainerDied","Data":"e787d8530b3567a8411d49571c3205e250ca554602f1525315156b82c2190629"}
Oct 30 18:38:05.828426 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:05.828390    2575 generic.go:356] "Generic (PLEG): container finished" podID="38cac2ec-1eff-4426-a43b-ebe89a226412" containerID="cb86f9482630b66d7bebedde4e94af5867883c41699f9db2876439f44acd7bc7" exitCode=0
Oct 30 18:38:05.828849 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:05.828434    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsgdd" event={"ID":"38cac2ec-1eff-4426-a43b-ebe89a226412","Type":"ContainerDied","Data":"cb86f9482630b66d7bebedde4e94af5867883c41699f9db2876439f44acd7bc7"}
Oct 30 18:38:06.638624 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:06.638590    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:06.638624 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:06.638623    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:06.638816 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:06.638697    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g92j9" podUID="f93cc289-68ea-4c64-9c47-0c16b1cf5f04"
Oct 30 18:38:06.638816 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:06.638729    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:06.638816 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:06.638805    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwbj7" podUID="d197c973-f7cd-4dff-b621-71fc64897eeb"
Oct 30 18:38:06.638918 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:06.638883    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7gn8" podUID="8dccf064-473f-461c-b188-19495b90f62c"
Oct 30 18:38:06.834050 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:06.833921    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 18:38:06.834842 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:06.834826    2575 scope.go:122] "RemoveContainer" containerID="edf1c2068373cb6c246d9093fc6f281031d75fd96acea08bea6c228d0e099909"
Oct 30 18:38:06.834842 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:06.834830    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv" event={"ID":"2a647480-88ca-47a2-9377-2841ce2e14f9","Type":"ContainerStarted","Data":"09984aca3808667febfd78301329d641c73ee1b92f81c238f9d3e1b56d1364fe"}
Oct 30 18:38:06.835057 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:06.834941    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:06.835285 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:06.835264    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:06.835416 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:06.835401    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:06.842681 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:06.842615    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsgdd" event={"ID":"38cac2ec-1eff-4426-a43b-ebe89a226412","Type":"ContainerStarted","Data":"e531154bbaadba637932cc3eff82d41c196d6e3555a51bfcd5d9771081f6cef5"}
Oct 30 18:38:06.852529 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:06.852406    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:06.855481 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:06.855456    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:07.184217 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:07.184096    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lsgdd" podStartSLOduration=9.167817975 podStartE2EDuration="22.184083556s" totalImagesPullingTime="13.016265581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:07.183796648 +0000 UTC m=+23.034830543" watchObservedRunningTime="2026-10-30 18:38:07.184083556 +0000 UTC m=+23.035117451"
Oct 30 18:38:07.848907 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:07.848878    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 18:38:07.849385 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:07.849238    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv" event={"ID":"2a647480-88ca-47a2-9377-2841ce2e14f9","Type":"ContainerStarted","Data":"ec2f3470fccbe24d51226778e2164f511f47287a8b0ce5d851505eb7fa41546f"}
Oct 30 18:38:08.027745 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:08.027689    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv" podStartSLOduration=7.664181651 podStartE2EDuration="23.027673927s" totalImagesPullingTime="15.363492276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:08.027206835 +0000 UTC m=+23.878240731" watchObservedRunningTime="2026-10-30 18:38:08.027673927 +0000 UTC m=+23.878707823"
Oct 30 18:38:08.638217 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:08.638191    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:08.638358 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:08.638238    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:08.638358 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:08.638253    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:08.638358 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:08.638343    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7gn8" podUID="8dccf064-473f-461c-b188-19495b90f62c"
Oct 30 18:38:08.638474 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:08.638382    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g92j9" podUID="f93cc289-68ea-4c64-9c47-0c16b1cf5f04"
Oct 30 18:38:08.638474 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:08.638461    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwbj7" podUID="d197c973-f7cd-4dff-b621-71fc64897eeb"
Oct 30 18:38:09.457795 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:09.457726    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s7gn8"]
Oct 30 18:38:09.458162 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:09.457829    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:09.458162 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:09.457912    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7gn8" podUID="8dccf064-473f-461c-b188-19495b90f62c"
Oct 30 18:38:09.460781 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:09.460761    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g92j9"]
Oct 30 18:38:09.460860 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:09.460835    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:09.460916 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:09.460902    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g92j9" podUID="f93cc289-68ea-4c64-9c47-0c16b1cf5f04"
Oct 30 18:38:09.461254 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:09.461230    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bwbj7"]
Oct 30 18:38:09.461325 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:09.461314    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:09.461396 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:09.461381    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwbj7" podUID="d197c973-f7cd-4dff-b621-71fc64897eeb"
Oct 30 18:38:10.638227 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:10.638196    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:10.638593 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:10.638337    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7gn8" podUID="8dccf064-473f-461c-b188-19495b90f62c"
Oct 30 18:38:11.639074 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:11.639045    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:11.639074 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:11.639082    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:11.639461 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:11.639175    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g92j9" podUID="f93cc289-68ea-4c64-9c47-0c16b1cf5f04"
Oct 30 18:38:11.639461 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:11.639303    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwbj7" podUID="d197c973-f7cd-4dff-b621-71fc64897eeb"
Oct 30 18:38:12.638825 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:12.638791    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:12.639070 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:12.638908    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7gn8" podUID="8dccf064-473f-461c-b188-19495b90f62c"
Oct 30 18:38:13.639236 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:13.639021    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:13.639658 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:13.639094    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:13.639658 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:13.639344    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwbj7" podUID="d197c973-f7cd-4dff-b621-71fc64897eeb"
Oct 30 18:38:13.639658 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:13.639415    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g92j9" podUID="f93cc289-68ea-4c64-9c47-0c16b1cf5f04"
Oct 30 18:38:14.639959 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:14.639928    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:14.640372 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:14.640051    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7gn8" podUID="8dccf064-473f-461c-b188-19495b90f62c"
Oct 30 18:38:15.398399 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.398375    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeReady"
Oct 30 18:38:15.398544 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.398494    2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 18:38:15.529247 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.529213    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-s6drh"]
Oct 30 18:38:15.533358 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.533338    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:15.580645 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.580622    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mhgzr"]
Oct 30 18:38:15.584242 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.584225    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s6drh"]
Oct 30 18:38:15.584339 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.584329    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mhgzr"
Oct 30 18:38:15.589316 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.589289    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:38:15.589316 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.589301    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vlkjn\""
Oct 30 18:38:15.589316 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.589304    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:38:15.634366 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.634341    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mhgzr"]
Oct 30 18:38:15.638908 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.638890    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:15.638965 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.638921    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:15.643916 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.643894    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwhqc\" (UniqueName: \"kubernetes.io/projected/ad954eaf-a685-4ac8-99f1-52f54994e109-kube-api-access-hwhqc\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:15.644345 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.643934    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad954eaf-a685-4ac8-99f1-52f54994e109-config-volume\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:15.644345 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.644003    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:15.644345 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.644109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad954eaf-a685-4ac8-99f1-52f54994e109-tmp-dir\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:15.689394 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.689367    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:38:15.689552 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.689450    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:38:15.689552 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.689481    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:38:15.689552 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.689519    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-64dnq\""
Oct 30 18:38:15.692474 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.692457    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:38:15.692752 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.692734    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:38:15.692807 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.692773    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:38:15.692899 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.692885    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dxlbz\""
Oct 30 18:38:15.692946 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.692923    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-l87lm\""
Oct 30 18:38:15.744819 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.744796    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/728e821c-4208-4b85-8b79-774997d92305-cert\") pod \"ingress-canary-mhgzr\" (UID: \"728e821c-4208-4b85-8b79-774997d92305\") " pod="openshift-ingress-canary/ingress-canary-mhgzr"
Oct 30 18:38:15.744908 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.744853    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwhqc\" (UniqueName: \"kubernetes.io/projected/ad954eaf-a685-4ac8-99f1-52f54994e109-kube-api-access-hwhqc\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:15.744961 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.744909    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad954eaf-a685-4ac8-99f1-52f54994e109-config-volume\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:15.744961 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.744943    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:15.745032 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.744970    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtngp\" (UniqueName: \"kubernetes.io/projected/728e821c-4208-4b85-8b79-774997d92305-kube-api-access-gtngp\") pod \"ingress-canary-mhgzr\" (UID: \"728e821c-4208-4b85-8b79-774997d92305\") " pod="openshift-ingress-canary/ingress-canary-mhgzr"
Oct 30 18:38:15.745032 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:15.745009    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:38:15.745032 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.745021    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad954eaf-a685-4ac8-99f1-52f54994e109-tmp-dir\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:15.745179 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:15.745056    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls podName:ad954eaf-a685-4ac8-99f1-52f54994e109 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:16.245040718 +0000 UTC m=+32.096074594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls") pod "dns-default-s6drh" (UID: "ad954eaf-a685-4ac8-99f1-52f54994e109") : secret "dns-default-metrics-tls" not found
Oct 30 18:38:15.745482 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.745467    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad954eaf-a685-4ac8-99f1-52f54994e109-tmp-dir\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:15.745773 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.745757    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad954eaf-a685-4ac8-99f1-52f54994e109-config-volume\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:15.809410 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.809376    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwhqc\" (UniqueName: \"kubernetes.io/projected/ad954eaf-a685-4ac8-99f1-52f54994e109-kube-api-access-hwhqc\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:15.845803 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.845770    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/728e821c-4208-4b85-8b79-774997d92305-cert\") pod \"ingress-canary-mhgzr\" (UID: \"728e821c-4208-4b85-8b79-774997d92305\") " pod="openshift-ingress-canary/ingress-canary-mhgzr"
Oct 30 18:38:15.845942 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.845852    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtngp\" (UniqueName: \"kubernetes.io/projected/728e821c-4208-4b85-8b79-774997d92305-kube-api-access-gtngp\") pod \"ingress-canary-mhgzr\" (UID: \"728e821c-4208-4b85-8b79-774997d92305\") " pod="openshift-ingress-canary/ingress-canary-mhgzr"
Oct 30 18:38:15.848047 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.848024    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/728e821c-4208-4b85-8b79-774997d92305-cert\") pod \"ingress-canary-mhgzr\" (UID: \"728e821c-4208-4b85-8b79-774997d92305\") " pod="openshift-ingress-canary/ingress-canary-mhgzr"
Oct 30 18:38:15.909058 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:15.909031    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtngp\" (UniqueName: \"kubernetes.io/projected/728e821c-4208-4b85-8b79-774997d92305-kube-api-access-gtngp\") pod \"ingress-canary-mhgzr\" (UID: \"728e821c-4208-4b85-8b79-774997d92305\") " pod="openshift-ingress-canary/ingress-canary-mhgzr"
Oct 30 18:38:16.195048 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:16.194970    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mhgzr"
Oct 30 18:38:16.250052 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:16.250028    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:16.250234 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:16.250210    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:38:16.250303 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:16.250292    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls podName:ad954eaf-a685-4ac8-99f1-52f54994e109 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:17.250272128 +0000 UTC m=+33.101306006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls") pod "dns-default-s6drh" (UID: "ad954eaf-a685-4ac8-99f1-52f54994e109") : secret "dns-default-metrics-tls" not found
Oct 30 18:38:16.610644 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:16.610614    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mhgzr"]
Oct 30 18:38:16.614044 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:38:16.614012    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728e821c_4208_4b85_8b79_774997d92305.slice/crio-8aa88336e4585e6830c791e18cb467b64a9fd9dadc6529c5ef484124f47825a4 WatchSource:0}: Error finding container 8aa88336e4585e6830c791e18cb467b64a9fd9dadc6529c5ef484124f47825a4: Status 404 returned error can't find the container with id 8aa88336e4585e6830c791e18cb467b64a9fd9dadc6529c5ef484124f47825a4
Oct 30 18:38:16.638789 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:16.638772    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:16.693033 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:16.693010    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:38:16.693349 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:16.693232    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-swx9z\""
Oct 30 18:38:16.870837 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:16.870762    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mhgzr" event={"ID":"728e821c-4208-4b85-8b79-774997d92305","Type":"ContainerStarted","Data":"8aa88336e4585e6830c791e18cb467b64a9fd9dadc6529c5ef484124f47825a4"}
Oct 30 18:38:17.256906 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:17.256821    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:17.257046 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:17.256939    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:38:17.257046 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:17.257001    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls podName:ad954eaf-a685-4ac8-99f1-52f54994e109 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:19.256988383 +0000 UTC m=+35.108022257 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls") pod "dns-default-s6drh" (UID: "ad954eaf-a685-4ac8-99f1-52f54994e109") : secret "dns-default-metrics-tls" not found
Oct 30 18:38:18.365866 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:18.365823    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:18.366297 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:18.365979    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:38:18.366297 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:18.366048    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs podName:8dccf064-473f-461c-b188-19495b90f62c nodeName:}" failed. No retries permitted until 2026-10-30 18:38:50.366029243 +0000 UTC m=+66.217063127 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs") pod "network-metrics-daemon-s7gn8" (UID: "8dccf064-473f-461c-b188-19495b90f62c") : secret "metrics-daemon-secret" not found
Oct 30 18:38:18.466362 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:18.466333    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:18.466549 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:18.466376    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sknx\" (UniqueName: \"kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx\") pod \"network-check-target-g92j9\" (UID: \"f93cc289-68ea-4c64-9c47-0c16b1cf5f04\") " pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:18.469375 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:18.469354    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sknx\" (UniqueName: \"kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx\") pod \"network-check-target-g92j9\" (UID: \"f93cc289-68ea-4c64-9c47-0c16b1cf5f04\") " pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:18.470479 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:18.470453    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:18.651596 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:18.651531    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:18.655700 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:18.655676    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:19.078439 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:19.078403    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g92j9"]
Oct 30 18:38:19.097107 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:38:19.097078    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93cc289_68ea_4c64_9c47_0c16b1cf5f04.slice/crio-36055232f4f584dc216a366a233fc1e68bb0a71309b67dd5e50242ea1b7e6773 WatchSource:0}: Error finding container 36055232f4f584dc216a366a233fc1e68bb0a71309b67dd5e50242ea1b7e6773: Status 404 returned error can't find the container with id 36055232f4f584dc216a366a233fc1e68bb0a71309b67dd5e50242ea1b7e6773
Oct 30 18:38:19.129678 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:19.129654    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bwbj7"]
Oct 30 18:38:19.133210 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:38:19.133188    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd197c973_f7cd_4dff_b621_71fc64897eeb.slice/crio-a3fc746b900bba43727c27aa6198d719b997633421f110d55a795f6ce32c74da WatchSource:0}: Error finding container a3fc746b900bba43727c27aa6198d719b997633421f110d55a795f6ce32c74da: Status 404 returned error can't find the container with id a3fc746b900bba43727c27aa6198d719b997633421f110d55a795f6ce32c74da
Oct 30 18:38:19.272739 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:19.272706    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:19.272925 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:19.272832    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:38:19.272925 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:19.272895    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls podName:ad954eaf-a685-4ac8-99f1-52f54994e109 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:23.272880762 +0000 UTC m=+39.123914651 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls") pod "dns-default-s6drh" (UID: "ad954eaf-a685-4ac8-99f1-52f54994e109") : secret "dns-default-metrics-tls" not found
Oct 30 18:38:19.879433 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:19.879397    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g92j9" event={"ID":"f93cc289-68ea-4c64-9c47-0c16b1cf5f04","Type":"ContainerStarted","Data":"36055232f4f584dc216a366a233fc1e68bb0a71309b67dd5e50242ea1b7e6773"}
Oct 30 18:38:19.880452 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:19.880432    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bwbj7" event={"ID":"d197c973-f7cd-4dff-b621-71fc64897eeb","Type":"ContainerStarted","Data":"a3fc746b900bba43727c27aa6198d719b997633421f110d55a795f6ce32c74da"}
Oct 30 18:38:20.885953 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:20.885914    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mhgzr" event={"ID":"728e821c-4208-4b85-8b79-774997d92305","Type":"ContainerStarted","Data":"d3d418764141d4f5d01583291d52d9cd2767c8f1de167fb3ee3618ee094eb106"}
Oct 30 18:38:21.625074 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.625019    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mhgzr" podStartSLOduration=3.178668678 podStartE2EDuration="6.625001216s" totalImagesPullingTime="3.446332538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:21.006319694 +0000 UTC m=+36.857353591" watchObservedRunningTime="2026-10-30 18:38:21.625001216 +0000 UTC m=+37.476035309"
Oct 30 18:38:21.625348 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.625332    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8jhgk"]
Oct 30 18:38:21.641247 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.641217    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:21.691508 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.691413    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8jhgk"]
Oct 30 18:38:21.780589 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.780556    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:38:21.780735 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.780664    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:38:21.780735 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.780669    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:38:21.780813 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.780752    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:38:21.781115 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.781097    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vvdpg\""
Oct 30 18:38:21.791055 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.791025    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:21.791196 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.791078    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-crio-socket\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:21.791196 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.791097    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:21.791317 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.791245    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-data-volume\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:21.791377 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.791310    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk6jm\" (UniqueName: \"kubernetes.io/projected/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-api-access-mk6jm\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:21.843693 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.843658    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-pd6n6-bmftf"]
Oct 30 18:38:21.867430 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.867402    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:21.892048 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.891980    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:21.892048 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.892032    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-crio-socket\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:21.892506 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.892058    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:21.892506 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.892103    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-data-volume\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:21.892506 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.892180    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk6jm\" (UniqueName: \"kubernetes.io/projected/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-api-access-mk6jm\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:21.892506 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:21.892453    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:38:21.892506 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:21.892510    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-rbac-proxy-cm podName:32718cc8-e0d1-40fb-bc79-5e3085c6ea3a nodeName:}" failed. No retries permitted until 2026-10-30 18:38:22.392494555 +0000 UTC m=+38.243528441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8jhgk" (UID: "32718cc8-e0d1-40fb-bc79-5e3085c6ea3a") : configmap "kube-rbac-proxy" not found
Oct 30 18:38:21.892859 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.892835    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-crio-socket\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:21.892958 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:21.892941    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:38:21.893069 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:21.892988    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls podName:32718cc8-e0d1-40fb-bc79-5e3085c6ea3a nodeName:}" failed. No retries permitted until 2026-10-30 18:38:22.392974171 +0000 UTC m=+38.244008062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8jhgk" (UID: "32718cc8-e0d1-40fb-bc79-5e3085c6ea3a") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:38:21.893069 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.893022    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-data-volume\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:21.901660 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.901425    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-pd6n6-bmftf"]
Oct 30 18:38:21.924004 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.923977    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-z8gkb\""
Oct 30 18:38:21.926572 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.926525    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 18:38:21.979411 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.979372    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 18:38:21.982427 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.982393    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk6jm\" (UniqueName: \"kubernetes.io/projected/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-api-access-mk6jm\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:21.993155 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.993064    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-service-ca-bundle\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:21.993155 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.993106    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-archives-path\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:21.993155 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.993153    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-trusted-ca-bundle\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:21.993394 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:21.993175    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spnb6\" (UniqueName: \"kubernetes.io/projected/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-kube-api-access-spnb6\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:22.094585 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.094547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-service-ca-bundle\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:22.094733 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.094594    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-archives-path\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:22.094733 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.094621    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-trusted-ca-bundle\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:22.094733 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.094637    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spnb6\" (UniqueName: \"kubernetes.io/projected/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-kube-api-access-spnb6\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:22.095385 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.095366    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-service-ca-bundle\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:22.095565 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.095552    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-archives-path\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:22.096270 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.096250    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-trusted-ca-bundle\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:22.120308 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.120278    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hqvpc"]
Oct 30 18:38:22.147189 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.146988    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hqvpc"
Oct 30 18:38:22.159480 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.159453    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spnb6\" (UniqueName: \"kubernetes.io/projected/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-kube-api-access-spnb6\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:22.177299 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.177272    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hqvpc"]
Oct 30 18:38:22.186263 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.186224    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:22.200871 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.200827    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 18:38:22.200871 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.200862    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 18:38:22.201038 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.200880    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mxwmp\""
Oct 30 18:38:22.296619 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.296588    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrsg8\" (UniqueName: \"kubernetes.io/projected/131610dd-0ebb-4f53-8339-e987b304d6f7-kube-api-access-vrsg8\") pod \"migrator-66df5d496b-hqvpc\" (UID: \"131610dd-0ebb-4f53-8339-e987b304d6f7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hqvpc"
Oct 30 18:38:22.397159 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.397112    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:22.397345 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.397189    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:22.397345 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:22.397237    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:38:22.397345 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:22.397302    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-rbac-proxy-cm podName:32718cc8-e0d1-40fb-bc79-5e3085c6ea3a nodeName:}" failed. No retries permitted until 2026-10-30 18:38:23.397286611 +0000 UTC m=+39.248320498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8jhgk" (UID: "32718cc8-e0d1-40fb-bc79-5e3085c6ea3a") : configmap "kube-rbac-proxy" not found
Oct 30 18:38:22.397345 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.397238    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrsg8\" (UniqueName: \"kubernetes.io/projected/131610dd-0ebb-4f53-8339-e987b304d6f7-kube-api-access-vrsg8\") pod \"migrator-66df5d496b-hqvpc\" (UID: \"131610dd-0ebb-4f53-8339-e987b304d6f7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hqvpc"
Oct 30 18:38:22.397345 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:22.397331    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:38:22.397611 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:22.397393    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls podName:32718cc8-e0d1-40fb-bc79-5e3085c6ea3a nodeName:}" failed. No retries permitted until 2026-10-30 18:38:23.397379671 +0000 UTC m=+39.248413575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8jhgk" (UID: "32718cc8-e0d1-40fb-bc79-5e3085c6ea3a") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:38:22.459977 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.459943    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrsg8\" (UniqueName: \"kubernetes.io/projected/131610dd-0ebb-4f53-8339-e987b304d6f7-kube-api-access-vrsg8\") pod \"migrator-66df5d496b-hqvpc\" (UID: \"131610dd-0ebb-4f53-8339-e987b304d6f7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hqvpc"
Oct 30 18:38:22.475893 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.475865    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hqvpc"
Oct 30 18:38:22.588159 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.587542    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-pd6n6-bmftf"]
Oct 30 18:38:22.590805 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:38:22.590771    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc8f47a_4f4c_4adf_9b0b_668536b61bc9.slice/crio-a8cd336d75bdecabfafc7e272d16c8038312bbea957c6c242b945f80a4e9c24f WatchSource:0}: Error finding container a8cd336d75bdecabfafc7e272d16c8038312bbea957c6c242b945f80a4e9c24f: Status 404 returned error can't find the container with id a8cd336d75bdecabfafc7e272d16c8038312bbea957c6c242b945f80a4e9c24f
Oct 30 18:38:22.786152 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.786047    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4z6kx"]
Oct 30 18:38:22.811936 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.811551    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx"
Oct 30 18:38:22.843312 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.843237    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4z6kx"]
Oct 30 18:38:22.866399 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.866373    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 18:38:22.866646 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.866608    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8jsgf\""
Oct 30 18:38:22.866718 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.866708    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 18:38:22.866786 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.866774    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 18:38:22.866821 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.866814    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 18:38:22.878149 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.877665    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hqvpc"]
Oct 30 18:38:22.883656 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:38:22.883606    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131610dd_0ebb_4f53_8339_e987b304d6f7.slice/crio-c4832be9d3e34f493c57016af6d3f2b90bbdb84a4a9b93a3c166df44c2d81fe5 WatchSource:0}: Error finding container c4832be9d3e34f493c57016af6d3f2b90bbdb84a4a9b93a3c166df44c2d81fe5: Status 404 returned error can't find the container with id c4832be9d3e34f493c57016af6d3f2b90bbdb84a4a9b93a3c166df44c2d81fe5
Oct 30 18:38:22.891453 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.891425    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pd6n6-bmftf" event={"ID":"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9","Type":"ContainerStarted","Data":"a8cd336d75bdecabfafc7e272d16c8038312bbea957c6c242b945f80a4e9c24f"}
Oct 30 18:38:22.892727 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.892700    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hqvpc" event={"ID":"131610dd-0ebb-4f53-8339-e987b304d6f7","Type":"ContainerStarted","Data":"c4832be9d3e34f493c57016af6d3f2b90bbdb84a4a9b93a3c166df44c2d81fe5"}
Oct 30 18:38:22.901119 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.901098    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a1fefd23-c919-4384-85c6-116295c18142-signing-cabundle\") pod \"service-ca-5fffc97bdf-4z6kx\" (UID: \"a1fefd23-c919-4384-85c6-116295c18142\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx"
Oct 30 18:38:22.901215 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.901147    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhkvt\" (UniqueName: \"kubernetes.io/projected/a1fefd23-c919-4384-85c6-116295c18142-kube-api-access-lhkvt\") pod \"service-ca-5fffc97bdf-4z6kx\" (UID: \"a1fefd23-c919-4384-85c6-116295c18142\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx"
Oct 30 18:38:22.901267 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:22.901235    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a1fefd23-c919-4384-85c6-116295c18142-signing-key\") pod \"service-ca-5fffc97bdf-4z6kx\" (UID: \"a1fefd23-c919-4384-85c6-116295c18142\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx"
Oct 30 18:38:23.002220 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:23.002181    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a1fefd23-c919-4384-85c6-116295c18142-signing-cabundle\") pod \"service-ca-5fffc97bdf-4z6kx\" (UID: \"a1fefd23-c919-4384-85c6-116295c18142\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx"
Oct 30 18:38:23.002374 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:23.002224    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkvt\" (UniqueName: \"kubernetes.io/projected/a1fefd23-c919-4384-85c6-116295c18142-kube-api-access-lhkvt\") pod \"service-ca-5fffc97bdf-4z6kx\" (UID: \"a1fefd23-c919-4384-85c6-116295c18142\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx"
Oct 30 18:38:23.002374 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:23.002259    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a1fefd23-c919-4384-85c6-116295c18142-signing-key\") pod \"service-ca-5fffc97bdf-4z6kx\" (UID: \"a1fefd23-c919-4384-85c6-116295c18142\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx"
Oct 30 18:38:23.002922 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:23.002901    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a1fefd23-c919-4384-85c6-116295c18142-signing-cabundle\") pod \"service-ca-5fffc97bdf-4z6kx\" (UID: \"a1fefd23-c919-4384-85c6-116295c18142\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx"
Oct 30 18:38:23.004966 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:23.004944    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a1fefd23-c919-4384-85c6-116295c18142-signing-key\") pod \"service-ca-5fffc97bdf-4z6kx\" (UID: \"a1fefd23-c919-4384-85c6-116295c18142\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx"
Oct 30 18:38:23.068093 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:23.068061    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkvt\" (UniqueName: \"kubernetes.io/projected/a1fefd23-c919-4384-85c6-116295c18142-kube-api-access-lhkvt\") pod \"service-ca-5fffc97bdf-4z6kx\" (UID: \"a1fefd23-c919-4384-85c6-116295c18142\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx"
Oct 30 18:38:23.136474 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:23.136442    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx"
Oct 30 18:38:23.305058 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:23.305027    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:23.305189 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:23.305175    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:38:23.305254 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:23.305242    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls podName:ad954eaf-a685-4ac8-99f1-52f54994e109 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:31.305222399 +0000 UTC m=+47.156256275 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls") pod "dns-default-s6drh" (UID: "ad954eaf-a685-4ac8-99f1-52f54994e109") : secret "dns-default-metrics-tls" not found
Oct 30 18:38:23.406118 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:23.406047    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:23.406329 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:23.406180    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:23.406329 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:23.406251    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:38:23.406329 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:23.406266    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:38:23.406329 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:23.406328    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls podName:32718cc8-e0d1-40fb-bc79-5e3085c6ea3a nodeName:}" failed. No retries permitted until 2026-10-30 18:38:25.406307866 +0000 UTC m=+41.257341755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8jhgk" (UID: "32718cc8-e0d1-40fb-bc79-5e3085c6ea3a") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:38:23.406525 ip-10-0-135-66 kubenswrapper[2575]: E1030 18:38:23.406349    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-rbac-proxy-cm podName:32718cc8-e0d1-40fb-bc79-5e3085c6ea3a nodeName:}" failed. No retries permitted until 2026-10-30 18:38:25.406338106 +0000 UTC m=+41.257371983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8jhgk" (UID: "32718cc8-e0d1-40fb-bc79-5e3085c6ea3a") : configmap "kube-rbac-proxy" not found
Oct 30 18:38:23.566038 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:23.566006    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4z6kx"]
Oct 30 18:38:23.566399 ip-10-0-135-66 kubenswrapper[2575]: W1030 18:38:23.566368    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1fefd23_c919_4384_85c6_116295c18142.slice/crio-2d3a95987c523d8f0b6e8ebe78ce11bdf3fcf13bb81ca3f35defddd6ad244c35 WatchSource:0}: Error finding container 2d3a95987c523d8f0b6e8ebe78ce11bdf3fcf13bb81ca3f35defddd6ad244c35: Status 404 returned error can't find the container with id 2d3a95987c523d8f0b6e8ebe78ce11bdf3fcf13bb81ca3f35defddd6ad244c35
Oct 30 18:38:23.898285 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:23.897950    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g92j9" event={"ID":"f93cc289-68ea-4c64-9c47-0c16b1cf5f04","Type":"ContainerStarted","Data":"ebe79a2bf301054530599b5e9996d0e95a5071da43ccd89f22cc50d35091a754"}
Oct 30 18:38:23.898285 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:23.898041    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:23.899576 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:23.899548    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx" event={"ID":"a1fefd23-c919-4384-85c6-116295c18142","Type":"ContainerStarted","Data":"2d3a95987c523d8f0b6e8ebe78ce11bdf3fcf13bb81ca3f35defddd6ad244c35"}
Oct 30 18:38:24.017932 ip-10-0-135-66 kubenswrapper[2575]: I1030 18:38:24.016922    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g92j9" podStartSLOduration=34.918727876 podStartE2EDuration="39.016905432s" totalImagesPullingTime="4.098177556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:24.016442486 +0000 UTC m=+39.867476381" watchObservedRunningTime="2026-10-30 18:38:24.016905432 +0000 UTC m=+39.867939329"
Oct 30 18:38:25.396825 ip-10-0-135-66 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 18:38:25.421644 ip-10-0-135-66 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 18:38:25.421876 ip-10-0-135-66 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 18:38:25.421940 ip-10-0-135-66 systemd[1]: kubelet.service: Consumed 2.390s CPU time.
Oct 30 18:38:25.431688 ip-10-0-135-66 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:38:25.555228 ip-10-0-135-66 kubenswrapper[4511]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:38:25.555228 ip-10-0-135-66 kubenswrapper[4511]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:38:25.555228 ip-10-0-135-66 kubenswrapper[4511]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:38:25.555228 ip-10-0-135-66 kubenswrapper[4511]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:38:25.558868 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558777    4511 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:25.558868 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558865    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:25.558868 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558868    4511 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:25.558868 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558872    4511 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:25.558868 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558875    4511 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558878    4511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558881    4511 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558884    4511 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558887    4511 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558890    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558892    4511 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558895    4511 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558897    4511 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558900    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558903    4511 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558905    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558908    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558912    4511 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558916    4511 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558926    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558931    4511 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558935    4511 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558939    4511 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:25.559112 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558943    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558947    4511 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558951    4511 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558955    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558961    4511 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558965    4511 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558970    4511 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558975    4511 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558979    4511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558982    4511 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558984    4511 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558987    4511 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558990    4511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558994    4511 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.558999    4511 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559004    4511 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559008    4511 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559012    4511 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559017    4511 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:25.559807 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559022    4511 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559027    4511 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559032    4511 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559035    4511 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559038    4511 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559041    4511 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559044    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559047    4511 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559050    4511 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559053    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559056    4511 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559058    4511 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559061    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559063    4511 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559066    4511 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559068    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559071    4511 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559076    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559080    4511 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559085    4511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:25.560507 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559089    4511 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559093    4511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559095    4511 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559098    4511 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559101    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559104    4511 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559107    4511 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559111    4511 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559116    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559120    4511 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559138    4511 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559143    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559147    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559150    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559153    4511 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559158    4511 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559162    4511 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559167    4511 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559170    4511 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:25.561178 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559174    4511 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559178    4511 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559183    4511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559187    4511 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559192    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559196    4511 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559200    4511 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559203    4511 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559205    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559209    4511 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559213    4511 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559218    4511 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559222    4511 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559227    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559232    4511 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559235    4511 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559238    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559241    4511 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559244    4511 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559246    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:25.561790 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559249    4511 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559252    4511 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559255    4511 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559258    4511 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.559261    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560332    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560342    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560347    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560352    4511 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560357    4511 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560361    4511 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560365    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560369    4511 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560373    4511 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560378    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560382    4511 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560386    4511 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560391    4511 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560395    4511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560399    4511 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:25.562530 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560404    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560409    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560412    4511 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560414    4511 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560417    4511 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560420    4511 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560422    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560426    4511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560429    4511 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560431    4511 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560434    4511 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560437    4511 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560439    4511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560442    4511 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560444    4511 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560447    4511 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560450    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560453    4511 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560456    4511 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:25.563340 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560458    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560461    4511 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560463    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560466    4511 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560470    4511 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560475    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560479    4511 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560484    4511 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560488    4511 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560492    4511 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560495    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560499    4511 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560505    4511 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560511    4511 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560516    4511 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560520    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560523    4511 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560529    4511 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560533    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560538    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:25.563940 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560542    4511 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560546    4511 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560550    4511 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560554    4511 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560559    4511 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560563    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560567    4511 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560571    4511 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560575    4511 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560579    4511 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560583    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560587    4511 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560591    4511 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560596    4511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560600    4511 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560605    4511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560607    4511 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560610    4511 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560613    4511 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560616    4511 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:25.564588 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560620    4511 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560622    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560625    4511 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560627    4511 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560630    4511 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560633    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560635    4511 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560638    4511 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560641    4511 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560643    4511 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560646    4511 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560649    4511 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560654    4511 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560658    4511 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560663    4511 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560667    4511 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560671    4511 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560675    4511 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560677    4511 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:25.565223 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560680    4511 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560682    4511 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560687    4511 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560690    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560693    4511 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560696    4511 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560698    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560701    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560703    4511 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560706    4511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560708    4511 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560711    4511 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.560716    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561405    4511 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561414    4511 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561418    4511 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561423    4511 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561427    4511 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561432    4511 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:25.565804 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561435    4511 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561438    4511 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561443    4511 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561448    4511 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561453    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561457    4511 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561461    4511 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561465    4511 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561469    4511 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561474    4511 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561480    4511 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561485    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561489    4511 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561494    4511 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561497    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561501    4511 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561506    4511 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561510    4511 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561514    4511 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561518    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:25.566379 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561522    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561526    4511 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561531    4511 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561535    4511 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561539    4511 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561544    4511 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561548    4511 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561553    4511 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561557    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561566    4511 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561573    4511 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561579    4511 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561583    4511 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561587    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561589    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561592    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561595    4511 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561599    4511 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561603    4511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:25.566975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561606    4511 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561609    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561612    4511 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561615    4511 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561617    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561620    4511 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561622    4511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561625    4511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561628    4511 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561633    4511 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561637    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561641    4511 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561645    4511 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561649    4511 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561653    4511 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561658    4511 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561661    4511 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561665    4511 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561669    4511 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:25.567593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561673    4511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561676    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561681    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561686    4511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561691    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561694    4511 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561697    4511 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561700    4511 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561702    4511 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561705    4511 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561708    4511 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561712    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561716    4511 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561720    4511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561724    4511 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561728    4511 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561733    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561738    4511 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561742    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:25.568110 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561746    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561751    4511 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561755    4511 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561760    4511 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561764    4511 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561769    4511 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561772    4511 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561775    4511 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561778    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561780    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561784    4511 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561786    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561789    4511 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561792    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561794    4511 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561797    4511 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561808    4511 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561812    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561818    4511 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:25.568767 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561822    4511 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561826    4511 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561830    4511 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.561835    4511 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.561951    4511 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.561973    4511 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.561984    4511 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.561992    4511 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.561999    4511 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562005    4511 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562012    4511 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562019    4511 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562025    4511 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562029    4511 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562035    4511 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562040    4511 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562046    4511 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562051    4511 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562055    4511 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562060    4511 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562064    4511 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562067    4511 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562072    4511 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562075    4511 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:38:25.569334 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562078    4511 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562081    4511 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562084    4511 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562088    4511 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562091    4511 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562095    4511 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562099    4511 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562106    4511 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562110    4511 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562112    4511 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562116    4511 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562118    4511 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562123    4511 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562142    4511 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562147    4511 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562151    4511 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562154    4511 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562157    4511 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562164    4511 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562169    4511 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562173    4511 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562178    4511 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562181    4511 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562186    4511 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:38:25.570283 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562189    4511 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562192    4511 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562194    4511 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562197    4511 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562200    4511 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562203    4511 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562207    4511 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562212    4511 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562216    4511 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562221    4511 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562228    4511 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562233    4511 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562237    4511 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562240    4511 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562243    4511 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562250    4511 flags.go:64] FLAG: --help="false"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562253    4511 flags.go:64] FLAG: --hostname-override="ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562256    4511 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562259    4511 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562262    4511 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562266    4511 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562269    4511 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562272    4511 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:38:25.571063 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562275    4511 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562279    4511 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562284    4511 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562290    4511 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562295    4511 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562300    4511 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562305    4511 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562308    4511 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562311    4511 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562314    4511 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562317    4511 flags.go:64] FLAG: --lock-file=""
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562320    4511 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562322    4511 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562325    4511 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562333    4511 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562338    4511 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562343    4511 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562348    4511 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562351    4511 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562354    4511 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562357    4511 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562361    4511 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562369    4511 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562374    4511 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562383    4511 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:38:25.571781 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562387    4511 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562391    4511 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562394    4511 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562397    4511 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562400    4511 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562402    4511 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562413    4511 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562426    4511 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562431    4511 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562436    4511 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562440    4511 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562446    4511 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562450    4511 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562457    4511 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562461    4511 flags.go:64] FLAG: --port="10250"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562467    4511 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562471    4511 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-04af32871652ebe57"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562476    4511 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562482    4511 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562487    4511 flags.go:64] FLAG: --register-node="true"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562492    4511 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562498    4511 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562503    4511 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562505    4511 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562508    4511 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562512    4511 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:38:25.572727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562515    4511 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562518    4511 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562523    4511 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562528    4511 flags.go:64] FLAG: --runonce="false"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562533    4511 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562542    4511 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562548    4511 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562555    4511 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562558    4511 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562562    4511 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562565    4511 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562568    4511 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562571    4511 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562573    4511 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562576    4511 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562580    4511 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562584    4511 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562589    4511 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562593    4511 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562602    4511 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562607    4511 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562611    4511 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562617    4511 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562621    4511 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562628    4511 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:38:25.573691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562632    4511 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562637    4511 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562641    4511 flags.go:64] FLAG: --v="2"
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562649    4511 flags.go:64] FLAG: --version="false"
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562655    4511 flags.go:64] FLAG: --vmodule=""
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562661    4511 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.562667    4511 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562853    4511 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562860    4511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562865    4511 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562870    4511 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562874    4511 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562880    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562884    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562888    4511 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562893    4511 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562897    4511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562901    4511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562905    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562910    4511 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562915    4511 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:25.574593 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562920    4511 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562925    4511 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562930    4511 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562934    4511 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562938    4511 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562941    4511 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562943    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562946    4511 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562949    4511 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562951    4511 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562958    4511 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562965    4511 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562972    4511 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562977    4511 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562982    4511 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562986    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562990    4511 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562994    4511 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.562998    4511 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:25.575334 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563002    4511 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563008    4511 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563011    4511 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563014    4511 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563019    4511 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563025    4511 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563029    4511 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563033    4511 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563039    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563043    4511 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563048    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563052    4511 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563056    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563060    4511 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563064    4511 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563068    4511 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563072    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563076    4511 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563080    4511 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563083    4511 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:25.576030 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563086    4511 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563088    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563091    4511 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563096    4511 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563100    4511 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563105    4511 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563109    4511 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563115    4511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563119    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563140    4511 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563145    4511 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563149    4511 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563153    4511 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563158    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563162    4511 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563167    4511 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563173    4511 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563176    4511 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563179    4511 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563181    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:25.576800 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563184    4511 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563194    4511 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563199    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563203    4511 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563207    4511 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563212    4511 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563215    4511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563218    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563221    4511 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563225    4511 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563230    4511 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563234    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563239    4511 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563244    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563249    4511 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563254    4511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563257    4511 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563259    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563262    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563265    4511 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:25.577543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563268    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:25.578309 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563272    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:25.578309 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563276    4511 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:25.578309 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563282    4511 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:25.578309 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563288    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:25.578309 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563294    4511 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:25.578309 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563299    4511 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:25.578309 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563304    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:25.578309 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563312    4511 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:25.578309 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563316    4511 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:25.578309 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563321    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:25.578309 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563325    4511 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:25.578309 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563328    4511 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:25.578309 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.563361    4511 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:38:25.578309 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563409    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:25.578309 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563416    4511 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:25.578309 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563420    4511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563423    4511 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563426    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563429    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563433    4511 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563438    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563442    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563446    4511 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563453    4511 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563459    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563463    4511 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563465    4511 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563470    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563473    4511 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563475    4511 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563478    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563481    4511 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563484    4511 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563486    4511 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563489    4511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:25.578889 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563491    4511 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563494    4511 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563496    4511 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563498    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563502    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563505    4511 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563507    4511 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563510    4511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563513    4511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563516    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563519    4511 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563525    4511 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563530    4511 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563535    4511 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563540    4511 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563544    4511 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563548    4511 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563550    4511 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563553    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:25.579483 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563556    4511 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563558    4511 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563561    4511 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563563    4511 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563569    4511 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563575    4511 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563578    4511 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563584    4511 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563589    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563593    4511 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563597    4511 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563602    4511 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563606    4511 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563611    4511 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563614    4511 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563616    4511 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563619    4511 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563622    4511 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563624    4511 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:25.580161 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563627    4511 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563630    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563633    4511 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563636    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563638    4511 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563641    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563648    4511 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563651    4511 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563653    4511 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563656    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563658    4511 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563661    4511 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563663    4511 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563666    4511 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563669    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563671    4511 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563673    4511 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563676    4511 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563680    4511 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:25.580867 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563683    4511 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563688    4511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563693    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563698    4511 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563703    4511 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563707    4511 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563711    4511 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563715    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563719    4511 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563721    4511 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563724    4511 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563726    4511 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563729    4511 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563732    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563736    4511 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563741    4511 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563745    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563749    4511 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563753    4511 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563757    4511 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:25.581543 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563768    4511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563772    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563776    4511 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563780    4511 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563785    4511 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563788    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.563791    4511 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.569007    4511 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.569023    4511 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.569044    4511 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.569051    4511 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569158    4511 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569166    4511 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569171    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569176    4511 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569181    4511 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569186    4511 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569191    4511 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569195    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569200    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:25.582154 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569204    4511 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569209    4511 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569213    4511 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569218    4511 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569222    4511 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569226    4511 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569230    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569232    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569235    4511 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569238    4511 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569241    4511 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569243    4511 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569246    4511 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569248    4511 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569251    4511 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569253    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569257    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569259    4511 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569262    4511 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:25.582760 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569265    4511 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569275    4511 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569278    4511 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569281    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569283    4511 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569286    4511 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569289    4511 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569291    4511 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569294    4511 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569297    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569299    4511 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569302    4511 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569304    4511 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569307    4511 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569309    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569312    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569314    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569317    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569319    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569322    4511 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:25.583499 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569324    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569327    4511 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569329    4511 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569332    4511 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569336    4511 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569341    4511 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569346    4511 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569351    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569355    4511 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569360    4511 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569365    4511 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569370    4511 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569374    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569378    4511 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569386    4511 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569389    4511 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569392    4511 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569394    4511 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569397    4511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569399    4511 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:25.584185 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569402    4511 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569405    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569408    4511 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569411    4511 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569413    4511 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569416    4511 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569418    4511 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569421    4511 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569423    4511 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569426    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569428    4511 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569431    4511 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569434    4511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569436    4511 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569439    4511 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569442    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569444    4511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569446    4511 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569449    4511 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:25.584811 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569451    4511 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569453    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569456    4511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569458    4511 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569462    4511 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569467    4511 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569472    4511 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569476    4511 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569490    4511 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569495    4511 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569500    4511 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569504    4511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569506    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569509    4511 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569511    4511 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569514    4511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569517    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569520    4511 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:25.585446 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569525    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:25.586053 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.569553    4511 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:38:25.586053 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569754    4511 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:25.586053 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569761    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:25.586053 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569766    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:25.586053 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569770    4511 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:25.586053 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569773    4511 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:25.586053 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569776    4511 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:25.586053 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569778    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:25.586053 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569781    4511 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:25.586053 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569784    4511 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:25.586053 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569786    4511 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:25.586053 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569789    4511 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:25.586053 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569791    4511 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:25.586053 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569794    4511 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:25.586053 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569797    4511 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:25.586053 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569799    4511 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569802    4511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569804    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569806    4511 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569809    4511 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569812    4511 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569814    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569824    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569827    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569831    4511 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569835    4511 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569837    4511 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569840    4511 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569842    4511 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569845    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569847    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569850    4511 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569852    4511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569855    4511 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:25.586647 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569857    4511 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569860    4511 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569862    4511 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569865    4511 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569867    4511 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569870    4511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569872    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569875    4511 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569877    4511 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569880    4511 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569882    4511 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569885    4511 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569889    4511 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569896    4511 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569901    4511 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569906    4511 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569910    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569914    4511 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569918    4511 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:25.587345 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569922    4511 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569925    4511 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569935    4511 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569938    4511 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569941    4511 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569943    4511 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569946    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569948    4511 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569951    4511 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569954    4511 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569956    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569959    4511 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569961    4511 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569964    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569966    4511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569969    4511 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569971    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569974    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569976    4511 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569978    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:25.588036 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569981    4511 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569983    4511 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569988    4511 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569991    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.569999    4511 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570002    4511 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570004    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570007    4511 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570009    4511 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570012    4511 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570016    4511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570020    4511 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570024    4511 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570027    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570029    4511 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570037    4511 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570040    4511 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570042    4511 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570045    4511 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570048    4511 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:25.588721 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570050    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:25.589525 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570053    4511 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:25.589525 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570055    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:25.589525 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570059    4511 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:25.589525 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570062    4511 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:25.589525 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570064    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:25.589525 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570067    4511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:25.589525 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570070    4511 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:25.589525 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570072    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:25.589525 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570075    4511 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:25.589525 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570077    4511 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:25.589525 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570080    4511 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:25.589525 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570083    4511 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:25.589525 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570085    4511 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:25.589525 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.570110    4511 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:38:25.589525 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570189    4511 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:25.589525 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570197    4511 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570202    4511 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570206    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570212    4511 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570217    4511 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570222    4511 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570227    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570231    4511 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570235    4511 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570237    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570240    4511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570243    4511 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570252    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570255    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570257    4511 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570260    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570265    4511 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570269    4511 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570273    4511 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:25.590066 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570278    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570284    4511 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570289    4511 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570294    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570298    4511 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570303    4511 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570307    4511 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570311    4511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570316    4511 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570319    4511 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570321    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570326    4511 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570328    4511 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570331    4511 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570334    4511 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570336    4511 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570339    4511 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570342    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570344    4511 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570347    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:25.590729 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570349    4511 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570352    4511 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570354    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570357    4511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570360    4511 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570362    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570371    4511 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570373    4511 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570377    4511 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570379    4511 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570382    4511 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570384    4511 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570387    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570390    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570393    4511 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570395    4511 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570397    4511 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570400    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570402    4511 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570413    4511 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:25.591447 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570416    4511 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570419    4511 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570421    4511 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570426    4511 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570428    4511 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570430    4511 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570433    4511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570435    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570438    4511 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570441    4511 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570443    4511 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570446    4511 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570448    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570450    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570454    4511 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570457    4511 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570460    4511 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570462    4511 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570465    4511 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:25.592159 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570474    4511 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570478    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570482    4511 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570486    4511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570489    4511 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570491    4511 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570494    4511 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570496    4511 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570499    4511 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570502    4511 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570504    4511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570507    4511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570509    4511 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570512    4511 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570514    4511 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570516    4511 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570520    4511 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570523    4511 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570528    4511 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:25.592773 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570532    4511 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:25.593491 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570535    4511 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:25.593491 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570539    4511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:25.593491 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570542    4511 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:25.593491 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570545    4511 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:25.593491 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570549    4511 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:25.593491 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570553    4511 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:25.593491 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:25.570557    4511 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:25.593491 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.570640    4511 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:38:25.593491 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.571821    4511 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 18:38:25.593491 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.571875    4511 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 18:38:25.593491 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.572110    4511 server.go:1014] "Starting client certificate rotation"
Oct 30 18:38:25.593491 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.572256    4511 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:38:25.593491 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.572400    4511 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:32:45 +0000 UTC" deadline="2028-06-29 22:10:41.218251292 +0000 UTC"
Oct 30 18:38:25.593491 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.572479    4511 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14595h32m15.645779001s"
Oct 30 18:38:25.593491 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.572864    4511 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:38:25.594000 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.574797    4511 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:38:25.594000 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.577634    4511 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:38:25.594000 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.579843    4511 log.go:25] "Validated CRI v1 image API"
Oct 30 18:38:25.594000 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.580680    4511 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:38:25.594000 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.582552    4511 fs.go:135] Filesystem UUIDs: map[50f16d56-ad77-4ad8-82bc-346a2e08680a:/dev/nvme0n1p3 51d14c8d-fdf7-4b84-a193-a00a66fe12de:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 18:38:25.594219 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.582574    4511 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2325f2ad51e14b7934614b25dc3c88bf2bf6a6027136849332fa220e958903fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2325f2ad51e14b7934614b25dc3c88bf2bf6a6027136849332fa220e958903fc/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27d3510732809f2f8ce65fec367e12425b2394c2fe693a5b390ebacf52247b13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27d3510732809f2f8ce65fec367e12425b2394c2fe693a5b390ebacf52247b13/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d3a95987c523d8f0b6e8ebe78ce11bdf3fcf13bb81ca3f35defddd6ad244c35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d3a95987c523d8f0b6e8ebe78ce11bdf3fcf13bb81ca3f35defddd6ad244c35/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36055232f4f584dc216a366a233fc1e68bb0a71309b67dd5e50242ea1b7e6773/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36055232f4f584dc216a366a233fc1e68bb0a71309b67dd5e50242ea1b7e6773/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/429f48f2a4437f4a2921d6230babddf6639f995add43b4c25f613eda43984e89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/429f48f2a4437f4a2921d6230babddf6639f995add43b4c25f613eda43984e89/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c4d929837c0848ffd43b672413822b449f17b7e429c0485e97902f24932bb36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c4d929837c0848ffd43b672413822b449f17b7e429c0485e97902f24932bb36/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8aa88336e4585e6830c791e18cb467b64a9fd9dadc6529c5ef484124f47825a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8aa88336e4585e6830c791e18cb467b64a9fd9dadc6529c5ef484124f47825a4/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ea384eecbbf339ff3da088dbeb1dcd0cd4559b14e8176236caa9ce344a36a01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ea384eecbbf339ff3da088dbeb1dcd0cd4559b14e8176236caa9ce344a36a01/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3fc746b900bba43727c27aa6198d719b997633421f110d55a795f6ce32c74da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3fc746b900bba43727c27aa6198d719b997633421f110d55a795f6ce32c74da/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a50cbce0de3a3d4ff53e77cedbf55d85b0bd76f421acd1dd22b3c3edb2171962/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a50cbce0de3a3d4ff53e77cedbf55d85b0bd76f421acd1dd22b3c3edb2171962/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6b8df729ef95c302ae3ad908c1a598848f2daa57e6b3f16555403a1e390805a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6b8df729ef95c302ae3ad908c1a598848f2daa57e6b3f16555403a1e390805a/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8cd336d75bdecabfafc7e272d16c8038312bbea957c6c242b945f80a4e9c24f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8cd336d75bdecabfafc7e272d16c8038312bbea957c6c242b945f80a4e9c24f/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/baa3f0a35e2d65e89e4bc0f526c85bf91ba4af35a01193febb924f5e9f91f644/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/baa3f0a35e2d65e89e4bc0f526c85bf91ba4af35a01193febb924f5e9f91f644/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4832be9d3e34f493c57016af6d3f2b90bbdb84a4a9b93a3c166df44c2d81fe5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4832be9d3e34f493c57016af6d3f2b90bbdb84a4a9b93a3c166df44c2d81fe5/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d74739a98b7763906ca579fafc2acb7039362b0f99d932c6a7c22e376ecc1f10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d74739a98b7763906ca579fafc2acb7039362b0f99d932c6a7c22e376ecc1f10/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f03e3791eff44f8f48ae35a5363d1ce851f7733d1d9c4df0c76b8e934e85aaf7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f03e3791eff44f8f48ae35a5363d1ce851f7733d1d9c4df0c76b8e934e85aaf7/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/131610dd-0ebb-4f53-8339-e987b304d6f7/volumes/kubernetes.io~projected/kube-api-access-vrsg8:{mountpoint:/var/lib/kubelet/pods/131610dd-0ebb-4f53-8339-e987b304d6f7/volumes/kubernetes.io~projected/kube-api-access-vrsg8 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/146371eb-2e54-477f-a7ba-c13450f54828/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/146371eb-2e54-477f-a7ba-c13450f54828/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/146371eb-2e54-477f-a7ba-c13450f54828/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/146371eb-2e54-477f-a7ba-c13450f54828/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/146371eb-2e54-477f-a7ba-c13450f54828/volumes/kubernetes.io~projected/kube-api-access-xjjll:{mountpoint:/var/lib/kubelet/pods/146371eb-2e54-477f-a7ba-c13450f54828/volumes/kubernetes.io~projected/kube-api-access-xjjll major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a647480-88ca-47a2-9377-2841ce2e14f9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2a647480-88ca-47a2-9377-2841ce2e14f9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a647480-88ca-47a2-9377-2841ce2e14f9/volumes/kubernetes.io~projected/kube-api-access-c7s62:{mountpoint:/var/lib/kubelet/pods/2a647480-88ca-47a2-9377-2841ce2e14f9/volumes/kubernetes.io~projected/kube-api-access-c7s62 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a647480-88ca-47a2-9377-2841ce2e14f9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2a647480-88ca-47a2-9377-2841ce2e14f9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a/volumes/kubernetes.io~projected/kube-api-access-mk6jm:{mountpoint:/var/lib/kubelet/pods/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a/volumes/kubernetes.io~projected/kube-api-access-mk6jm major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38cac2ec-1eff-4426-a43b-ebe89a226412/volumes/kubernetes.io~projected/kube-api-access-5nspw:{mountpoint:/var/lib/kubelet/pods/38cac2ec-1eff-4426-a43b-ebe89a226412/volumes/kubernetes.io~projected/kube-api-access-5nspw major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b6f291a-fb97-4253-b3f1-39d0d482ca88/volumes/kubernetes.io~projected/kube-api-access-w2l6s:{mountpoint:/var/lib/kubelet/pods/5b6f291a-fb97-4253-b3f1-39d0d482ca88/volumes/kubernetes.io~projected/kube-api-access-w2l6s major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9/volumes/kubernetes.io~projected/kube-api-access-spnb6:{mountpoint:/var/lib/kubelet/pods/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9/volumes/kubernetes.io~projected/kube-api-access-spnb6 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/728e821c-4208-4b85-8b79-774997d92305/volumes/kubernetes.io~projected/kube-api-access-gtngp:{mountpoint:/var/lib/kubelet/pods/728e821c-4208-4b85-8b79-774997d92305/volumes/kubernetes.io~projected/kube-api-access-gtngp major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/728e821c-4208-4b85-8b79-774997d92305/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/728e821c-4208-4b85-8b79-774997d92305/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8dccf064-473f-461c-b188-19495b90f62c/volumes/kubernetes.io~projected/kube-api-access-9gsgj:{mountpoint:/var/lib/kubelet/pods/8dccf064-473f-461c-b188-19495b90f62c/volumes/kubernetes.io~projected/kube-api-access-9gsgj major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9defd701-4377-47c0-bfaa-a298d0acc866/volumes/kubernetes.io~projected/kube-api-access-zw99q:{mountpoint:/var/lib/kubelet/pods/9defd701-4377-47c0-bfaa-a298d0acc866/volumes/kubernetes.io~projected/kube-api-access-zw99q major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1fefd23-c919-4384-85c6-116295c18142/volumes/kubernetes.io~projected/kube-api-access-lhkvt:{mountpoint:/var/lib/kubelet/pods/a1fefd23-c919-4384-85c6-116295c18142/volumes/kubernetes.io~projected/kube-api-access-lhkvt major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1fefd23-c919-4384-85c6-116295c18142/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/a1fefd23-c919-4384-85c6-116295c18142/volumes/kubernetes.io~secret/signing-key major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad954eaf-a685-4ac8-99f1-52f54994e109/volumes/kubernetes.io~projected/kube-api-access-hwhqc:{mountpoint:/var/lib/kubelet/pods/ad954eaf-a685-4ac8-99f1-52f54994e109/volumes/kubernetes.io~projected/kube-api-access-hwhqc major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3/volumes/kubernetes.io~projected/kube-api-access-78zp5:{mountpoint:/var/lib/kubelet/pods/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3/volumes/kubernetes.io~projected/kube-api-access-78zp5 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbf4cda5-bce1-417d-94db-213431b7dd0e/volumes/kubernetes.io~projected/kube-api-access-rnb4p:{mountpoint:/var/lib/kubelet/pods/cbf4cda5-bce1-417d-94db-213431b7dd0e/volumes/kubernetes.io~projected/kube-api-access-rnb4p major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce2cd81e-eece-495b-98d8-73fc05aef957/volumes/kubernetes.io~projected/kube-api-access-7pgvf:{mountpoint:/var/lib/kubelet/pods/ce2cd81e-eece-495b-98d8-73fc05aef957/volumes/kubernetes.io~projected/kube-api-access-7pgvf major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d197c973-f7cd-4dff-b621-71fc64897eeb/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d197c973-f7cd-4dff-b621-71fc64897eeb/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f93cc289-68ea-4c64-9c47-0c16b1cf5f04/volumes/kubernetes.io~projected/kube-api-access-6sknx:{mountpoint:/var/lib/kubelet/pods/f93cc289-68ea-4c64-9c47-0c16b1cf5f04/volumes/kubernetes.io~projected/kube-api-access-6sknx major:0 minor:206 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/7d6ffbbe5628645fc9237cfa1a6dc7a94d5a9c25f52688f99bd249c94def157c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e0f5aa2e443edbd9405a0fbc400f06f60bac63878cba6de07841e4bd71e006e7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1135f50cd2ca93c4b4fc91a4a117549adf38f477d42a66e849db4c3d50ed779b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/9c7f073f0c3231195232169557c26a2b7e9b07014f626696736687d485648bf6/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/14729dbff4addb44782c0edd492fa65a8cd360c18940bac87c849d758c80f999/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b8b339fa0b7d0111bd616b0367420db2b7c10349b373a4a63bf10039772b0dd3/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/95a967c0d274c6db15e2601deb7c5d762a2ee99ee41303e1ab3fd2a56f91f40b/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2b19e7465257d44ba03d2b12b7477a0d98c385be7a754992b8c6175c8c6e1458/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3b1e8da72fe20705f36ebbd4e041de48298049caffeb631dd823b98e65d810d7/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/d835a8914dc5f393771ac59df33ede3919786905c58ed7288deddf1b43038beb/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/efe236c8176097cb6e73cd7dc126b889063256981730953867f44c1486ea46c6/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/de3a3eea611992b8c9c04518de914e96199f18bbd8c02d11ed4b958c3db6727f/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ce1f0c391eaff256233391f3affe1d9443fa67ffd1836265c8ee7be73d6813f4/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/a73503dfb04e89ea2f07f8630ebf669025b3054d2b20f4a4a279c52300f28247/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/2c91f88b8e9119fb51205c85e581478e250b5097c47a5dca5b794078f80b1627/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/52e3a1388e407925ca9a0071a642284f46fbfbe17f5c6480fc3994980f736e6c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/37f1a9d1d1ce1b115c7627adf96d4103e0c70bd3d7b740abd6ebcfd96de79902/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d2f417f66c8cf539b6402caa016aceb4bc7651cbdb856fc30d9c291a65aeac65/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4210ae216730cbf61fd2ecee64a61d3a9a35d88660ea5522211db1cfcdca9732/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c0211c0579c253ab816cd737c79d42b42fc487e10df5e41194a82e60df9d6539/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/08e108b2546ddecbb8959dbb4933a5ed303167b0513c43bb8e4e5637620d4751/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/550ccd0be8457aeec3a9f9db71fad8b848f519f9a7710d3b4201bb2312d94b4a/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/784d32bead7f397c814a47305893ff9184ee3b04d0b0e1c561fc9eaac03a8318/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/84526a7e78e80bd4dab56719c21b1af03dc6661dccbff7507078b7a12b2c1095/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/9aa953d0eb806bb77120c7d13a19f4c9344cfbfb4282a66c2b4fdfc791ccdb99/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/466a65aecb7b3a4c47096e4b513a316e264b64752fa4b7cb91f91cba86a82911/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/4199d538ad98cc881dacac13d1d67be8e530152c001df1657ea02fcdfd9588ad/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/41553ab6e8dc6fee5514317a405ebdb8d36b28e252d8c80810e65f18201d5150/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/99f146e6fab5dbb4e1f4cfebbd87c10b219545155229029159dbc44027c3ce06/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/ab29036b9b356c9b9d88a970463abca546838bd56308c9ccef348deff2fdf61d/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/323225d12c7877a1318953b8c215e37226833f2a7b1cb0ad712ac902da81a982/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/1ceca4d19efb97a7c2f795940d78d74015ed2d660aa5f60c1c08fd8e97bde042/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/2d7d8b8af86cabcd2f1ce80eaf263cc21b6c2b66d7f9a3e29097720a423ab1b7/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/14438edb6eda4f29b909f58e2e83bd4340c06a06ac8af7c85fa7681ee42db3d1/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/f00c7aab24fa0bb2994b8ed3e3128946ebd7d308e43c58727e97e0105cc4b2e3/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/90a551fc73b082bd6e09d2e96961b7627ee1362b0d3a241903c6c7024b6d895f/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/513bd7ece9349f6165f730486b4077b68a45c897545763bdcbdb1e0409e858a4/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/0359ffc8373936127e0884cff4e465a0711a1fe6a5fa9a67d2f8ae0b9a4a7846/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/9bfb9b4b01e27209b1e0f3800a51c7ca40b5076af35f4253a0eb2ff7133276ed/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/ba0e2c8d33f939814433d2baf52609bd0996f58ea42fca965cfb9479d95b3832/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/88867f80ecef6a427ec324a396cffc93b2907865a6c5be2093889fef6f69fce7/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.590280    4511 manager.go:217] Machine: {Timestamp:2026-10-30 18:38:25.589655157 +0000 UTC m=+0.107340047 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099540 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec237ca8c704693c9d8e1ef58fd1cce6 SystemUUID:ec237ca8-c704-693c-9d8e-1ef58fd1cce6 BootID:c9ca2a92-618e-44af-b460-17642c895cce Filesystems:[{Device:/var/lib/kubelet/pods/5b6f291a-fb97-4253-b3f1-39d0d482ca88/volumes/kubernetes.io~projected/kube-api-access-w2l6s DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8aa88336e4585e6830c791e18cb467b64a9fd9dadc6529c5ef484124f47825a4/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f93cc289-68ea-4c64-9c47-0c16b1cf5f04/volumes/kubernetes.io~projected/kube-api-access-6sknx DeviceMajor:0 DeviceMinor:206 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36055232f4f584dc216a366a233fc1e68bb0a71309b67dd5e50242ea1b7e6773/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3/volumes/kubernetes.io~projected/kube-api-access-78zp5 DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/429f48f2a4437f4a2921d6230babddf6639f995add43b4c25f613eda43984e89/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad954eaf-a685-4ac8-99f1-52f54994e109/volumes/kubernetes.io~projected/kube-api-access-hwhqc DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9/volumes/kubernetes.io~projected/kube-api-access-spnb6 DeviceMajor:0 DeviceMinor:223 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbf4cda5-bce1-417d-94db-213431b7dd0e/volumes/kubernetes.io~projected/kube-api-access-rnb4p DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/146371eb-2e54-477f-a7ba-c13450f54828/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ea384eecbbf339ff3da088dbeb1dcd0cd4559b14e8176236caa9ce344a36a01/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/baa3f0a35e2d65e89e4bc0f526c85bf91ba4af35a01193febb924f5e9f91f644/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8dccf064-473f-461c-b188-19495b90f62c/volumes/kubernetes.io~projected/kube-api-access-9gsgj DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/728e821c-4208-4b85-8b79-774997d92305/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a647480-88ca-47a2-9377-2841ce2e14f9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2a647480-88ca-47a2-9377-2841ce2e14f9/volumes/kubernetes.io~projected/kube-api-access-c7s62 DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a/volumes/kubernetes.io~projected/kube-api-access-mk6jm DeviceMajor:0 DeviceMinor:218 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4832be9d3e34f493c57016af6d3f2b90bbdb84a4a9b93a3c166df44c2d81fe5/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a1fefd23-c919-4384-85c6-116295c18142/volumes/kubernetes.io~projected/kube-api-access-lhkvt DeviceMajor:0 DeviceMinor:234 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c4d929837c0848ffd43b672413822b449f17b7e429c0485e97902f24932bb36/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a50cbce0de3a3d4ff53e77cedbf55d85b0bd76f421acd1dd22b3c3edb2171962/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/146371eb-2e54-477f-a7ba-c13450f54828/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/728e821c-4208-4b85-8b79-774997d92305/volumes/kubernetes.io~projected/kube-api-access-gtngp DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/38cac2ec-1eff-4426-a43b-ebe89a226412/volumes/kubernetes.io~projected/kube-api-access-5nspw DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d3a95987c523d8f0b6e8ebe78ce11bdf3fcf13bb81ca3f35defddd6ad244c35/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d74739a98b7763906ca579fafc2acb7039362b0f99d932c6a7c22e376ecc1f10/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8cd336d75bdecabfafc7e272d16c8038312bbea957c6c242b945f80a4e9c24f/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a1fefd23-c919-4384-85c6-116295c18142/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:233 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6b8df729ef95c302ae3ad908c1a598848f2daa57e6b3f16555403a1e390805a/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a647480-88ca-47a2-9377-2841ce2e14f9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/131610dd-0ebb-4f53-8339-e987b304d6f7/volumes/kubernetes.io~projected/kube-api-access-vrsg8 DeviceMajor:0 DeviceMinor:226 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ce2cd81e-eece-495b-98d8-73fc05aef957/volumes/kubernetes.io~projected/kube-api-access-7pgvf DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9defd701-4377-47c0-bfaa-a298d0acc866/volumes/kubernetes.io~projected/kube-api-access-zw99q DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3fc746b900bba43727c27aa6198d719b997633421f110d55a795f6ce32c74da/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2325f2ad51e14b7934614b25dc3c88bf2bf6a6027136849332fa220e958903fc/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f03e3791eff44f8f48ae35a5363d1ce851f7733d1d9c4df0c76b8e934e85aaf7/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27d3510732809f2f8ce65fec367e12425b2394c2fe693a5b390ebacf52247b13/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d197c973-f7cd-4dff-b621-71fc64897eeb/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:207 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/146371eb-2e54-477f-a7ba-c13450f54828/volumes/kubernetes.io~projected/kube-api-access-xjjll DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2d3a95987c523d8 MacAddress:52:39:d1:41:b0:62 Speed:10000 Mtu:8901} {Name:36055232f4f584d MacAddress:7a:8f:16:fa:1b:89 Speed:10000 Mtu:8901} {Name:8aa88336e4585e6 MacAddress:c6:e9:f3:b9:68:03 Speed:10000 Mtu:8901} {Name:a3fc746b900bba4 MacAddress:42:ee:53:f7:32:92 Speed:10000 Mtu:8901} {Name:a8cd336d75bdeca MacAddress:ee:be:84:81:45:b6 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:e0:f5:e3:ed Speed:0 Mtu:9001} {Name:br-int MacAddress:6a:a5:73:00:c6:5f Speed:0 Mtu:8901} {Name:c4832be9d3e34f4 MacAddress:8e:84:69:f9:14:a2 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:e0:f5:e3:ed Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:d2:63:25:97:9c:f7 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:82:75:e8:b2:f6:f1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.590628    4511 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.590676    4511 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.590989    4511 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.591016    4511 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-66.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.591206    4511 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.591214    4511 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.591226    4511 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.591242    4511 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.591329    4511 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.591513    4511 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.591638    4511 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.591684    4511 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.591702    4511 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.591725    4511 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.591734    4511 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.591748    4511 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.592526    4511 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.592539    4511 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:38:25.594410 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.593008    4511 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:38:25.605546 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.605527    4511 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:38:25.605546 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.605544    4511 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:38:25.605698 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.605660    4511 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 18:38:25.605927 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.605906    4511 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:38:25.606027 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.605954    4511 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:38:25.606099 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.606055    4511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:38:25.606099 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.606074    4511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:38:25.606099 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.606082    4511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:38:25.606099 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.606092    4511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:38:25.606298 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.606103    4511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:38:25.606298 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.606114    4511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:38:25.606298 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.606139    4511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:38:25.606298 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.606150    4511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:38:25.606298 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.606162    4511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:38:25.606298 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.606171    4511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:38:25.606298 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.606182    4511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:38:25.606298 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.606192    4511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:38:25.606298 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.606219    4511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:38:25.606298 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.606229    4511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:38:25.610247 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.610225    4511 server.go:1263] "Started kubelet"
Oct 30 18:38:25.613714 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.610436    4511 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:38:25.613714 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.610526    4511 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:38:25.613714 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.611366    4511 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:38:25.613714 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.611814    4511 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:38:25.610735 ip-10-0-135-66 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:38:25.619566 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.617692    4511 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:38:25.619566 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.618047    4511 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:38:25.619566 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.618218    4511 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:38:25.619566 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.618593    4511 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:32:47 +0000 UTC" deadline="2028-04-22 20:03:13.786286437 +0000 UTC"
Oct 30 18:38:25.619566 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.618626    4511 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12961h24m48.167663s"
Oct 30 18:38:25.619566 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.618789    4511 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:38:25.619566 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.618802    4511 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:38:25.619566 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.619191    4511 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:38:25.619566 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:25.619368    4511 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found"
Oct 30 18:38:25.620849 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.620834    4511 factory.go:153] Registering CRI-O factory
Oct 30 18:38:25.620849 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.620852    4511 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:38:25.621001 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.620906    4511 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:38:25.621001 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.620916    4511 factory.go:55] Registering systemd factory
Oct 30 18:38:25.621001 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.620924    4511 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:38:25.621001 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.620953    4511 factory.go:103] Registering Raw factory
Oct 30 18:38:25.621001 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.620969    4511 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:38:25.621451 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.621438    4511 manager.go:319] Starting recovery of all containers
Oct 30 18:38:25.622042 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.621986    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b" volumeName="kubernetes.io/secret/5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b-agent-certs" seLinuxMountContext=""
Oct 30 18:38:25.622116 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622053    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad954eaf-a685-4ac8-99f1-52f54994e109" volumeName="kubernetes.io/empty-dir/ad954eaf-a685-4ac8-99f1-52f54994e109-tmp-dir" seLinuxMountContext=""
Oct 30 18:38:25.622116 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622068    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad954eaf-a685-4ac8-99f1-52f54994e109" volumeName="kubernetes.io/projected/ad954eaf-a685-4ac8-99f1-52f54994e109-kube-api-access-hwhqc" seLinuxMountContext=""
Oct 30 18:38:25.622116 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622081    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce2cd81e-eece-495b-98d8-73fc05aef957" volumeName="kubernetes.io/configmap/ce2cd81e-eece-495b-98d8-73fc05aef957-serviceca" seLinuxMountContext=""
Oct 30 18:38:25.622116 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622092    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce2cd81e-eece-495b-98d8-73fc05aef957" volumeName="kubernetes.io/projected/ce2cd81e-eece-495b-98d8-73fc05aef957-kube-api-access-7pgvf" seLinuxMountContext=""
Oct 30 18:38:25.622116 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622104    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a647480-88ca-47a2-9377-2841ce2e14f9" volumeName="kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-ovnkube-config" seLinuxMountContext=""
Oct 30 18:38:25.622116 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622116    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32718cc8-e0d1-40fb-bc79-5e3085c6ea3a" volumeName="kubernetes.io/projected/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-api-access-mk6jm" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622145    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b6f291a-fb97-4253-b3f1-39d0d482ca88" volumeName="kubernetes.io/configmap/5b6f291a-fb97-4253-b3f1-39d0d482ca88-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622159    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cc8f47a-4f4c-4adf-9b0b-668536b61bc9" volumeName="kubernetes.io/projected/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-kube-api-access-spnb6" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622171    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="728e821c-4208-4b85-8b79-774997d92305" volumeName="kubernetes.io/secret/728e821c-4208-4b85-8b79-774997d92305-cert" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622182    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9defd701-4377-47c0-bfaa-a298d0acc866" volumeName="kubernetes.io/projected/9defd701-4377-47c0-bfaa-a298d0acc866-kube-api-access-zw99q" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622193    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1fefd23-c919-4384-85c6-116295c18142" volumeName="kubernetes.io/secret/a1fefd23-c919-4384-85c6-116295c18142-signing-key" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622206    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad954eaf-a685-4ac8-99f1-52f54994e109" volumeName="kubernetes.io/configmap/ad954eaf-a685-4ac8-99f1-52f54994e109-config-volume" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622254    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a647480-88ca-47a2-9377-2841ce2e14f9" volumeName="kubernetes.io/projected/2a647480-88ca-47a2-9377-2841ce2e14f9-kube-api-access-c7s62" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622283    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b6f291a-fb97-4253-b3f1-39d0d482ca88" volumeName="kubernetes.io/projected/5b6f291a-fb97-4253-b3f1-39d0d482ca88-kube-api-access-w2l6s" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622294    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cc8f47a-4f4c-4adf-9b0b-668536b61bc9" volumeName="kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-service-ca-bundle" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622307    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d197c973-f7cd-4dff-b621-71fc64897eeb" volumeName="kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622320    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f93cc289-68ea-4c64-9c47-0c16b1cf5f04" volumeName="kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622333    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b" volumeName="kubernetes.io/configmap/5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b-konnectivity-ca" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622344    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cc8f47a-4f4c-4adf-9b0b-668536b61bc9" volumeName="kubernetes.io/empty-dir/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-archives-path" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622355    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbf4cda5-bce1-417d-94db-213431b7dd0e" volumeName="kubernetes.io/projected/cbf4cda5-bce1-417d-94db-213431b7dd0e-kube-api-access-rnb4p" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622367    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="131610dd-0ebb-4f53-8339-e987b304d6f7" volumeName="kubernetes.io/projected/131610dd-0ebb-4f53-8339-e987b304d6f7-kube-api-access-vrsg8" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622380    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38cac2ec-1eff-4426-a43b-ebe89a226412" volumeName="kubernetes.io/projected/38cac2ec-1eff-4426-a43b-ebe89a226412-kube-api-access-5nspw" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622390    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b6f291a-fb97-4253-b3f1-39d0d482ca88" volumeName="kubernetes.io/configmap/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-daemon-config" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622425    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="728e821c-4208-4b85-8b79-774997d92305" volumeName="kubernetes.io/projected/728e821c-4208-4b85-8b79-774997d92305-kube-api-access-gtngp" seLinuxMountContext=""
Oct 30 18:38:25.622435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622442    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1fefd23-c919-4384-85c6-116295c18142" volumeName="kubernetes.io/configmap/a1fefd23-c919-4384-85c6-116295c18142-signing-cabundle" seLinuxMountContext=""
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622455    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1b3ccb2-2cc4-4291-a39f-e2edfff638f3" volumeName="kubernetes.io/empty-dir/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-tmp-dir" seLinuxMountContext=""
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622467    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146371eb-2e54-477f-a7ba-c13450f54828" volumeName="kubernetes.io/empty-dir/146371eb-2e54-477f-a7ba-c13450f54828-etc-tuned" seLinuxMountContext=""
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622484    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32718cc8-e0d1-40fb-bc79-5e3085c6ea3a" volumeName="kubernetes.io/empty-dir/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-data-volume" seLinuxMountContext=""
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622509    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38cac2ec-1eff-4426-a43b-ebe89a226412" volumeName="kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622527    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cc8f47a-4f4c-4adf-9b0b-668536b61bc9" volumeName="kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622540    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1b3ccb2-2cc4-4291-a39f-e2edfff638f3" volumeName="kubernetes.io/projected/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-kube-api-access-78zp5" seLinuxMountContext=""
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622554    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146371eb-2e54-477f-a7ba-c13450f54828" volumeName="kubernetes.io/empty-dir/146371eb-2e54-477f-a7ba-c13450f54828-tmp" seLinuxMountContext=""
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622576    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a647480-88ca-47a2-9377-2841ce2e14f9" volumeName="kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622589    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38cac2ec-1eff-4426-a43b-ebe89a226412" volumeName="kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622601    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a647480-88ca-47a2-9377-2841ce2e14f9" volumeName="kubernetes.io/secret/2a647480-88ca-47a2-9377-2841ce2e14f9-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622615    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dccf064-473f-461c-b188-19495b90f62c" volumeName="kubernetes.io/projected/8dccf064-473f-461c-b188-19495b90f62c-kube-api-access-9gsgj" seLinuxMountContext=""
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622625    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9defd701-4377-47c0-bfaa-a298d0acc866" volumeName="kubernetes.io/configmap/9defd701-4377-47c0-bfaa-a298d0acc866-iptables-alerter-script" seLinuxMountContext=""
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622636    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1fefd23-c919-4384-85c6-116295c18142" volumeName="kubernetes.io/projected/a1fefd23-c919-4384-85c6-116295c18142-kube-api-access-lhkvt" seLinuxMountContext=""
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622649    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146371eb-2e54-477f-a7ba-c13450f54828" volumeName="kubernetes.io/projected/146371eb-2e54-477f-a7ba-c13450f54828-kube-api-access-xjjll" seLinuxMountContext=""
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622660    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a647480-88ca-47a2-9377-2841ce2e14f9" volumeName="kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-env-overrides" seLinuxMountContext=""
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622671    4511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38cac2ec-1eff-4426-a43b-ebe89a226412" volumeName="kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622682    4511 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:38:25.623380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.622690    4511 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:38:25.630813 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.630335    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 18:38:25.631094 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.631077    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 18:38:25.648932 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.648829    4511 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:38:25.650427 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.650395    4511 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:38:25.650427 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.650427    4511 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:38:25.650582 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.650448    4511 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:38:25.650582 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:25.650489    4511 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 18:38:25.669202 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.669122    4511 generic.go:356] "Generic (PLEG): container finished" podID="0b5fcda9dfeca1f860cdbf0f9fc64e83" containerID="8addf35385b5858e334e824bb2105c5237f40b2b4fdbb755523f62095a88a827" exitCode=0
Oct 30 18:38:25.691380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.691356    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 18:38:25.691933 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.691896    4511 generic.go:356] "Generic (PLEG): container finished" podID="2a647480-88ca-47a2-9377-2841ce2e14f9" containerID="edf1c2068373cb6c246d9093fc6f281031d75fd96acea08bea6c228d0e099909" exitCode=1
Oct 30 18:38:25.698039 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.698017    4511 generic.go:356] "Generic (PLEG): container finished" podID="38cac2ec-1eff-4426-a43b-ebe89a226412" containerID="cb86f9482630b66d7bebedde4e94af5867883c41699f9db2876439f44acd7bc7" exitCode=0
Oct 30 18:38:25.698039 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.698038    4511 generic.go:356] "Generic (PLEG): container finished" podID="38cac2ec-1eff-4426-a43b-ebe89a226412" containerID="e787d8530b3567a8411d49571c3205e250ca554602f1525315156b82c2190629" exitCode=0
Oct 30 18:38:25.698237 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.698046    4511 generic.go:356] "Generic (PLEG): container finished" podID="38cac2ec-1eff-4426-a43b-ebe89a226412" containerID="7b159e8ee3084f0bff30ad8faa7f3688b3a885716fd4904b4c00235f25496b64" exitCode=0
Oct 30 18:38:25.698237 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.698054    4511 generic.go:356] "Generic (PLEG): container finished" podID="38cac2ec-1eff-4426-a43b-ebe89a226412" containerID="65e19c8151db95d6589f1f4c74b3a0a99da838bf1772596388d185bfbda6990c" exitCode=0
Oct 30 18:38:25.698237 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.698061    4511 generic.go:356] "Generic (PLEG): container finished" podID="38cac2ec-1eff-4426-a43b-ebe89a226412" containerID="2498eeaeeca6982e97df4e9516a2e32acca3e6b6d708d75e4bd25aaca807ab97" exitCode=0
Oct 30 18:38:25.698237 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.698068    4511 generic.go:356] "Generic (PLEG): container finished" podID="38cac2ec-1eff-4426-a43b-ebe89a226412" containerID="78b3c6f252b6ae372501530983fb7235880d2e4a319ae550d861dc34d30c2d00" exitCode=0
Oct 30 18:38:25.700176 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.700158    4511 manager.go:324] Recovery completed
Oct 30 18:38:25.714333 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.714318    4511 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:25.715430 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.715414    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:25.715510 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.715448    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:25.715510 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.715462    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:25.716857 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.716841    4511 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:38:25.716857 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.716853    4511 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:25.716961 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.716857    4511 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:38:25.716961 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.716925    4511 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:38:25.717104 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.717087    4511 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 18:38:25.717176 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.717106    4511 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 18:38:25.717176 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.717148    4511 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 18:38:25.717176 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.717158    4511 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 18:38:25.717176 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.717165    4511 policy_none.go:50] "Start"
Oct 30 18:38:25.717360 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.717180    4511 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:38:25.717360 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.717194    4511 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:38:25.717454 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.717377    4511 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 18:38:25.717454 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.717388    4511 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 18:38:25.717454 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.717396    4511 policy_none.go:44] "Start"
Oct 30 18:38:25.717655 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.717640    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:25.717704 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.717668    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:25.717704 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.717678    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:25.720238 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:25.720212    4511 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found"
Oct 30 18:38:25.723103 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.723078    4511 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:38:25.725536 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.725522    4511 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:38:25.725615 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:25.725543    4511 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:38:25.725615 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.725551    4511 server.go:87] "Starting device plugin registration server"
Oct 30 18:38:25.725792 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.725779    4511 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:38:25.725833 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.725793    4511 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:38:25.725951 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.725938    4511 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:38:25.726056 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.726023    4511 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:38:25.726155 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.726064    4511 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:38:25.726155 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.726071    4511 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:38:25.726246 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.726145    4511 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:38:25.726054254Z","UUID":"16fcd9c4-23ed-4e58-b65a-11f463576818","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:38:25.727191 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.727172    4511 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:38:25.727267 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.727208    4511 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:38:25.727448 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:25.727427    4511 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:38:25.727570 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:25.727557    4511 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-66.us-west-2.compute.internal\" not found"
Oct 30 18:38:25.750763 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.750727    4511 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal"]
Oct 30 18:38:25.750852 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.750834    4511 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:25.751605 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.751591    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:25.751664 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.751618    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:25.751664 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.751628    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:25.751865 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.751854    4511 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:25.752014 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.751999    4511 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:25.752661 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.752643    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:25.752742 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.752663    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:25.752742 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.752679    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:25.752742 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.752687    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:25.752742 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.752694    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:25.752742 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.752698    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:25.752885 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:25.752822    4511 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found" node="ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:25.752922 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.752898    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal" event={"ID":"0b5fcda9dfeca1f860cdbf0f9fc64e83","Type":"ContainerStarted","Data":"12e2dac573cbbb37e34628cfb3db2e4f22d4b87706280f9f1c747f3176097c1c"}
Oct 30 18:38:25.752955 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.752930    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal" event={"ID":"0b5fcda9dfeca1f860cdbf0f9fc64e83","Type":"ContainerDied","Data":"8addf35385b5858e334e824bb2105c5237f40b2b4fdbb755523f62095a88a827"}
Oct 30 18:38:25.752955 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.752945    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal" event={"ID":"0b5fcda9dfeca1f860cdbf0f9fc64e83","Type":"ContainerStarted","Data":"a50cbce0de3a3d4ff53e77cedbf55d85b0bd76f421acd1dd22b3c3edb2171962"}
Oct 30 18:38:25.753007 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.752959    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal" event={"ID":"80e81f6072b8791c05f541935e51a65e","Type":"ContainerStarted","Data":"772dfee152cbb1e473cf868308c663f022b4f35b695c2ef9c13d2d5ca341fd65"}
Oct 30 18:38:25.753007 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.752970    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal" event={"ID":"80e81f6072b8791c05f541935e51a65e","Type":"ContainerStarted","Data":"6c4d929837c0848ffd43b672413822b449f17b7e429c0485e97902f24932bb36"}
Oct 30 18:38:25.753007 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.753000    4511 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:25.753675 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.753659    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:25.753675 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.753687    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:25.753831 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.753701    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:25.753884 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:25.753839    4511 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-66.us-west-2.compute.internal\" not found" node="ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:25.793397 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.793371    4511 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:25.793397 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.793387    4511 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:25.793539 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.793414    4511 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 18:38:25.826512 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.826492    4511 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:25.827432 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.827417    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:25.827520 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.827444    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:25.827520 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.827454    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:25.827520 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.827485    4511 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:25.839752 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.839737    4511 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:25.841782 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.841766    4511 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:25.893531 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.893508    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b5fcda9dfeca1f860cdbf0f9fc64e83-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"0b5fcda9dfeca1f860cdbf0f9fc64e83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:25.893615 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.893534    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b5fcda9dfeca1f860cdbf0f9fc64e83-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"0b5fcda9dfeca1f860cdbf0f9fc64e83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:25.893615 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.893561    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/80e81f6072b8791c05f541935e51a65e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"80e81f6072b8791c05f541935e51a65e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:25.919999 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.919946    4511 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:25.933407 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.933387    4511 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:25.933465 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.933439    4511 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:25.933945 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.933925    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeNotReady"
Oct 30 18:38:25.933997 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.933951    4511 setters.go:546] "Node became not ready" node="ip-10-0-135-66.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T18:38:25Z","lastTransitionTime":"2026-10-30T18:38:25Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 18:38:25.951214 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.951191    4511 nodeinfomanager.go:471] existing CSINode "ip-10-0-135-66.us-west-2.compute.internal" is owned by different node (oldNodeID="5e77d5eb-6b55-442f-9f8b-07edf9baa5a1", newNodeID=""), cleaning up...
Oct 30 18:38:25.981310 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.981290    4511 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:38:25.981392 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:25.981327    4511 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:25.981392 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.981339    4511 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:25.994554 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.994535    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b5fcda9dfeca1f860cdbf0f9fc64e83-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"0b5fcda9dfeca1f860cdbf0f9fc64e83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:25.994620 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.994566    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b5fcda9dfeca1f860cdbf0f9fc64e83-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"0b5fcda9dfeca1f860cdbf0f9fc64e83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:25.994620 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.994584    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/80e81f6072b8791c05f541935e51a65e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"80e81f6072b8791c05f541935e51a65e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:25.994717 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.994627    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/80e81f6072b8791c05f541935e51a65e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"80e81f6072b8791c05f541935e51a65e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:25.994717 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.994637    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b5fcda9dfeca1f860cdbf0f9fc64e83-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"0b5fcda9dfeca1f860cdbf0f9fc64e83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:25.994717 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:25.994663    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b5fcda9dfeca1f860cdbf0f9fc64e83-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal\" (UID: \"0b5fcda9dfeca1f860cdbf0f9fc64e83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:26.037320 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.037297    4511 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:38:26.037393 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:26.037335    4511 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:26.200350 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.200278    4511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-66.us-west-2.compute.internal" event="NodeReady"
Oct 30 18:38:26.200546 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.200365    4511 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 18:38:26.592718 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.592685    4511 apiserver.go:52] "Watching apiserver"
Oct 30 18:38:26.692938 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.692909    4511 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:38:26.693406 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.693386    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-pd6n6-bmftf","openshift-network-diagnostics/network-check-target-g92j9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5","openshift-ingress-canary/ingress-canary-mhgzr","kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-s7gn8","kube-system/global-pull-secret-syncer-bwbj7","openshift-image-registry/node-ca-5lhq5","openshift-multus/multus-additional-cni-plugins-lsgdd","openshift-multus/multus-jnnbn","kube-system/konnectivity-agent-dvpdt","openshift-dns/node-resolver-k4r8f","openshift-kube-storage-version-migrator/migrator-66df5d496b-hqvpc","openshift-dns/dns-default-s6drh","openshift-network-operator/iptables-alerter-gsknn","openshift-cluster-node-tuning-operator/tuned-f2b5l","openshift-insights/insights-runtime-extractor-8jhgk","openshift-ovn-kubernetes/ovnkube-node-n5qjv","openshift-service-ca/service-ca-5fffc97bdf-4z6kx"]
Oct 30 18:38:26.695840 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.695814    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:26.698111 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.698093    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:26.699546 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.699527    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:26.706032 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.706018    4511 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:26.706511 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.706491    4511 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:26.720251 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.720232    4511 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:38:26.744633 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.744607    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5c5lj\""
Oct 30 18:38:26.744785 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.744761    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vpq5z\""
Oct 30 18:38:26.746418 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.746394    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:38:26.746540 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.746528    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:38:26.746790 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.746774    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:38:26.747741 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.747726    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:38:26.747741 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.747734    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:38:26.747832 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.747811    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:38:26.747897 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.747884    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:38:26.747998 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.747984    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:38:26.748058 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.747981    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:38:26.748105 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.747982    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:38:26.748105 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.747999    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:38:26.748105 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.748018    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:38:26.748105 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.748099    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:38:26.748340 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.748318    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:38:26.748340 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.748307    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:38:26.748340 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.748344    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:38:26.748644 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.748324    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:38:26.748644 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.748324    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:38:26.748644 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.748350    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:38:26.748644 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.748368    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:38:26.748644 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.748389    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:38:26.748644 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.748395    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:38:26.748937 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.748675    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:38:26.748937 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.748778    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:38:26.748937 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.748848    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:38:26.748937 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.748908    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:38:26.749111 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.748969    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:38:26.749111 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.749051    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:38:26.798572 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.798547    4511 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:38:26.798651 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:26.798593    4511 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:26.799167 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799152    4511 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:38:26.799203 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:26.799193    4511 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-66.us-west-2.compute.internal"
Oct 30 18:38:26.799278 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799263    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad954eaf-a685-4ac8-99f1-52f54994e109-config-volume\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:26.799313 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799296    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-systemd\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.799341 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799323    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-env-overrides\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.799425 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799412    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-netns\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.799528 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799509    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-env-overrides\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.799567 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799515    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-kubelet\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.799597 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799581    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a1fefd23-c919-4384-85c6-116295c18142-signing-key\") pod \"service-ca-5fffc97bdf-4z6kx\" (UID: \"a1fefd23-c919-4384-85c6-116295c18142\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx"
Oct 30 18:38:26.799597 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799519    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad954eaf-a685-4ac8-99f1-52f54994e109-config-volume\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:26.799652 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799610    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-crio-socket\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:26.799652 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799630    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-cnibin\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:38:26.799652 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799645    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-tmp-dir\") pod \"node-resolver-k4r8f\" (UID: \"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3\") " pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:38:26.799744 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799663    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78zp5\" (UniqueName: \"kubernetes.io/projected/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-kube-api-access-78zp5\") pod \"node-resolver-k4r8f\" (UID: \"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3\") " pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:38:26.799744 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799687    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d197c973-f7cd-4dff-b621-71fc64897eeb-kubelet-config\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:26.799744 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799719    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sknx\" (UniqueName: \"kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx\") pod \"network-check-target-g92j9\" (UID: \"f93cc289-68ea-4c64-9c47-0c16b1cf5f04\") " pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:26.799744 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799739    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-tmp-dir\") pod \"node-resolver-k4r8f\" (UID: \"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3\") " pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:38:26.799896 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799741    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:26.799896 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799772    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-archives-path\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:26.799896 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799810    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nspw\" (UniqueName: \"kubernetes.io/projected/38cac2ec-1eff-4426-a43b-ebe89a226412-kube-api-access-5nspw\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:38:26.799896 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799861    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9defd701-4377-47c0-bfaa-a298d0acc866-iptables-alerter-script\") pod \"iptables-alerter-gsknn\" (UID: \"9defd701-4377-47c0-bfaa-a298d0acc866\") " pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:38:26.799896 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799887    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b6f291a-fb97-4253-b3f1-39d0d482ca88-cni-binary-copy\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.800117 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799910    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwhqc\" (UniqueName: \"kubernetes.io/projected/ad954eaf-a685-4ac8-99f1-52f54994e109-kube-api-access-hwhqc\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:26.800117 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799924    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-archives-path\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:26.800117 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799935    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7s62\" (UniqueName: \"kubernetes.io/projected/2a647480-88ca-47a2-9377-2841ce2e14f9-kube-api-access-c7s62\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.800117 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.799991    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysconfig\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.800117 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800009    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-cni-multus\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.800117 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800040    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-socket-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.800117 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800072    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/728e821c-4208-4b85-8b79-774997d92305-cert\") pod \"ingress-canary-mhgzr\" (UID: \"728e821c-4208-4b85-8b79-774997d92305\") " pod="openshift-ingress-canary/ingress-canary-mhgzr"
Oct 30 18:38:26.800394 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800173    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-cni-netd\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.800394 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800204    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9defd701-4377-47c0-bfaa-a298d0acc866-iptables-alerter-script\") pod \"iptables-alerter-gsknn\" (UID: \"9defd701-4377-47c0-bfaa-a298d0acc866\") " pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:38:26.800394 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800203    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysctl-d\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.800394 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800258    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysctl-conf\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.800394 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800257    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b6f291a-fb97-4253-b3f1-39d0d482ca88-cni-binary-copy\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.800394 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800308    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-var-lib-kubelet\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.800394 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800326    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-system-cni-dir\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:38:26.800394 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800373    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-cni-binary-copy\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:38:26.800662 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800408    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:38:26.800662 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800449    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spnb6\" (UniqueName: \"kubernetes.io/projected/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-kube-api-access-spnb6\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:26.800662 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800490    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-data-volume\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:26.800662 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800516    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-kubelet\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.800662 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800581    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-var-lib-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.800662 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800589    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-data-volume\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:26.800662 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800593    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-cni-binary-copy\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:38:26.800946 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800672    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-sys\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.800946 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800768    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:38:26.800946 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800796    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-host\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.800946 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800819    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:38:26.800946 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800836    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtngp\" (UniqueName: \"kubernetes.io/projected/728e821c-4208-4b85-8b79-774997d92305-kube-api-access-gtngp\") pod \"ingress-canary-mhgzr\" (UID: \"728e821c-4208-4b85-8b79-774997d92305\") " pod="openshift-ingress-canary/ingress-canary-mhgzr"
Oct 30 18:38:26.800946 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800867    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-systemd-units\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.800946 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800888    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-log-socket\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.801249 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800951    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-ovnkube-script-lib\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.801249 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.800979    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/146371eb-2e54-477f-a7ba-c13450f54828-tmp\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.801249 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801037    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-hosts-file\") pod \"node-resolver-k4r8f\" (UID: \"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3\") " pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:38:26.801249 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801057    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/146371eb-2e54-477f-a7ba-c13450f54828-tmp\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.801249 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801080    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38cac2ec-1eff-4426-a43b-ebe89a226412-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:38:26.801249 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801090    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9defd701-4377-47c0-bfaa-a298d0acc866-host-slash\") pod \"iptables-alerter-gsknn\" (UID: \"9defd701-4377-47c0-bfaa-a298d0acc866\") " pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:38:26.801249 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801108    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-socket-dir-parent\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.801249 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801157    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gsgj\" (UniqueName: \"kubernetes.io/projected/8dccf064-473f-461c-b188-19495b90f62c-kube-api-access-9gsgj\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:26.801249 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801174    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-cni-bin\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.801249 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801201    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-ovnkube-script-lib\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.801249 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801205    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-conf-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.801249 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801246    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-etc-kubernetes\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.801623 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801289    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2l6s\" (UniqueName: \"kubernetes.io/projected/5b6f291a-fb97-4253-b3f1-39d0d482ca88-kube-api-access-w2l6s\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.801623 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801341    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.801623 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801379    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-registration-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.801623 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801422    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-device-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.801623 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801441    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:26.801623 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801461    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrsg8\" (UniqueName: \"kubernetes.io/projected/131610dd-0ebb-4f53-8339-e987b304d6f7-kube-api-access-vrsg8\") pod \"migrator-66df5d496b-hqvpc\" (UID: \"131610dd-0ebb-4f53-8339-e987b304d6f7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hqvpc"
Oct 30 18:38:26.801623 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801492    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.801623 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801579    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.801623 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801604    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-kubernetes\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.801894 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801641    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjjll\" (UniqueName: \"kubernetes.io/projected/146371eb-2e54-477f-a7ba-c13450f54828-kube-api-access-xjjll\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.801894 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801664    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:38:26.801894 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801715    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-cnibin\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.801894 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801746    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-service-ca-bundle\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:26.801894 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801777    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a647480-88ca-47a2-9377-2841ce2e14f9-ovn-node-metrics-cert\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.801894 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801859    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw99q\" (UniqueName: \"kubernetes.io/projected/9defd701-4377-47c0-bfaa-a298d0acc866-kube-api-access-zw99q\") pod \"iptables-alerter-gsknn\" (UID: \"9defd701-4377-47c0-bfaa-a298d0acc866\") " pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:38:26.802070 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801924    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-system-cni-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.802070 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801943    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-cni-bin\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.802070 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801958    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.802070 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801976    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnb4p\" (UniqueName: \"kubernetes.io/projected/cbf4cda5-bce1-417d-94db-213431b7dd0e-kube-api-access-rnb4p\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.802070 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.801990    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce2cd81e-eece-495b-98d8-73fc05aef957-host\") pod \"node-ca-5lhq5\" (UID: \"ce2cd81e-eece-495b-98d8-73fc05aef957\") " pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:38:26.802070 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802039    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad954eaf-a685-4ac8-99f1-52f54994e109-tmp-dir\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:26.802262 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802078    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a1fefd23-c919-4384-85c6-116295c18142-signing-cabundle\") pod \"service-ca-5fffc97bdf-4z6kx\" (UID: \"a1fefd23-c919-4384-85c6-116295c18142\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx"
Oct 30 18:38:26.802262 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802147    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad954eaf-a685-4ac8-99f1-52f54994e109-tmp-dir\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:26.802262 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802180    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkvt\" (UniqueName: \"kubernetes.io/projected/a1fefd23-c919-4384-85c6-116295c18142-kube-api-access-lhkvt\") pod \"service-ca-5fffc97bdf-4z6kx\" (UID: \"a1fefd23-c919-4384-85c6-116295c18142\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx"
Oct 30 18:38:26.802262 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802215    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk6jm\" (UniqueName: \"kubernetes.io/projected/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-api-access-mk6jm\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:26.802262 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802232    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-run-netns\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.802419 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802306    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-node-log\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.802419 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802330    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-cni-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.802419 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802346    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-sys-fs\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.802419 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802368    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-systemd\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.802419 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802399    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-k8s-cni-cncf-io\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.802570 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802433    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b-agent-certs\") pod \"konnectivity-agent-dvpdt\" (UID: \"5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b\") " pod="kube-system/konnectivity-agent-dvpdt"
Oct 30 18:38:26.802570 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802509    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-hostroot\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.802570 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802542    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce2cd81e-eece-495b-98d8-73fc05aef957-serviceca\") pod \"node-ca-5lhq5\" (UID: \"ce2cd81e-eece-495b-98d8-73fc05aef957\") " pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:38:26.802659 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802569    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgvf\" (UniqueName: \"kubernetes.io/projected/ce2cd81e-eece-495b-98d8-73fc05aef957-kube-api-access-7pgvf\") pod \"node-ca-5lhq5\" (UID: \"ce2cd81e-eece-495b-98d8-73fc05aef957\") " pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:38:26.802714 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802700    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:26.802767 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802756    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-daemon-config\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.802805 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802765    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce2cd81e-eece-495b-98d8-73fc05aef957-serviceca\") pod \"node-ca-5lhq5\" (UID: \"ce2cd81e-eece-495b-98d8-73fc05aef957\") " pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:38:26.802805 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802796    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:26.802872 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802814    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/146371eb-2e54-477f-a7ba-c13450f54828-etc-tuned\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.802872 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802830    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-slash\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.802872 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802846    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.802872 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802861    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-modprobe-d\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.802985 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802878    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-multus-certs\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.802985 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802926    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/146371eb-2e54-477f-a7ba-c13450f54828-etc-tuned\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.802985 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802940    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-daemon-config\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.802985 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.802978    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b-konnectivity-ca\") pod \"konnectivity-agent-dvpdt\" (UID: \"5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b\") " pod="kube-system/konnectivity-agent-dvpdt"
Oct 30 18:38:26.803094 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.803015    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-lib-modules\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.803094 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.803054    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-trusted-ca-bundle\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:26.803168 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.803114    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:26.803168 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.803159    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-etc-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.803227 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.803203    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-ovn\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.803255 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.803231    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b-konnectivity-ca\") pod \"konnectivity-agent-dvpdt\" (UID: \"5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b\") " pod="kube-system/konnectivity-agent-dvpdt"
Oct 30 18:38:26.803255 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.803233    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-ovnkube-config\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.803310 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.803296    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d197c973-f7cd-4dff-b621-71fc64897eeb-dbus\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:26.803341 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.803321    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-run\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.803420 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.803406    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a647480-88ca-47a2-9377-2841ce2e14f9-ovnkube-config\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.844710 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.844658    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:38:26.845088 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845069    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mxwmp\""
Oct 30 18:38:26.845194 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845169    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:38:26.845194 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845177    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:38:26.845337 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845321    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s6x86\""
Oct 30 18:38:26.845461 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845448    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:38:26.845669 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845653    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:38:26.845733 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845716    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x4gd2\""
Oct 30 18:38:26.845733 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845724    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:38:26.845820 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845736    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-64dnq\""
Oct 30 18:38:26.845820 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845742    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 18:38:26.845820 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845736    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6fhn6\""
Oct 30 18:38:26.845820 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845737    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:38:26.845820 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845776    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dxlbz\""
Oct 30 18:38:26.845820 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845788    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:38:26.845820 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845790    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vvdpg\""
Oct 30 18:38:26.845820 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845806    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-rrn4s\""
Oct 30 18:38:26.846064 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845857    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w46g4\""
Oct 30 18:38:26.846064 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845865    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vlkjn\""
Oct 30 18:38:26.846064 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845865    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-z8gkb\""
Oct 30 18:38:26.846064 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845888    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-27m54\""
Oct 30 18:38:26.846064 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.845917    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-86nv8\""
Oct 30 18:38:26.846284 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.846174    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 18:38:26.846284 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.846184    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-swx9z\""
Oct 30 18:38:26.846284 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.846196    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:38:26.846284 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.846217    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:38:26.846284 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.846223    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 18:38:26.846284 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.846248    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 18:38:26.846284 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.846260    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-l87lm\""
Oct 30 18:38:26.846284 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.846274    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:38:26.846581 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.846324    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8jsgf\""
Oct 30 18:38:26.846581 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.846344    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:38:26.846581 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.846371    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 18:38:26.846581 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.846375    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 18:38:26.846581 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.846524    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 18:38:26.850302 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.850287    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a1fefd23-c919-4384-85c6-116295c18142-signing-key\") pod \"service-ca-5fffc97bdf-4z6kx\" (UID: \"a1fefd23-c919-4384-85c6-116295c18142\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx"
Oct 30 18:38:26.850358 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.850299    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/728e821c-4208-4b85-8b79-774997d92305-cert\") pod \"ingress-canary-mhgzr\" (UID: \"728e821c-4208-4b85-8b79-774997d92305\") " pod="openshift-ingress-canary/ingress-canary-mhgzr"
Oct 30 18:38:26.852433 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.852412    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-service-ca-bundle\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:26.852483 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.852470    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a647480-88ca-47a2-9377-2841ce2e14f9-ovn-node-metrics-cert\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.852518 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.852504    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a1fefd23-c919-4384-85c6-116295c18142-signing-cabundle\") pod \"service-ca-5fffc97bdf-4z6kx\" (UID: \"a1fefd23-c919-4384-85c6-116295c18142\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx"
Oct 30 18:38:26.852619 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.852607    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b-agent-certs\") pod \"konnectivity-agent-dvpdt\" (UID: \"5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b\") " pod="kube-system/konnectivity-agent-dvpdt"
Oct 30 18:38:26.853651 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.853639    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d197c973-f7cd-4dff-b621-71fc64897eeb-original-pull-secret\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:26.894622 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.893982    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 18:38:26.894622 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.894020    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwhqc\" (UniqueName: \"kubernetes.io/projected/ad954eaf-a685-4ac8-99f1-52f54994e109-kube-api-access-hwhqc\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:26.894622 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.894081    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7s62\" (UniqueName: \"kubernetes.io/projected/2a647480-88ca-47a2-9377-2841ce2e14f9-kube-api-access-c7s62\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.894622 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.894169    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78zp5\" (UniqueName: \"kubernetes.io/projected/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-kube-api-access-78zp5\") pod \"node-resolver-k4r8f\" (UID: \"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3\") " pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:38:26.894622 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.894242    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gsgj\" (UniqueName: \"kubernetes.io/projected/8dccf064-473f-461c-b188-19495b90f62c-kube-api-access-9gsgj\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:26.894622 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.894279    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnb4p\" (UniqueName: \"kubernetes.io/projected/cbf4cda5-bce1-417d-94db-213431b7dd0e-kube-api-access-rnb4p\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.894622 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.894293    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk6jm\" (UniqueName: \"kubernetes.io/projected/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-api-access-mk6jm\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:26.894622 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.894303    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkvt\" (UniqueName: \"kubernetes.io/projected/a1fefd23-c919-4384-85c6-116295c18142-kube-api-access-lhkvt\") pod \"service-ca-5fffc97bdf-4z6kx\" (UID: \"a1fefd23-c919-4384-85c6-116295c18142\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx"
Oct 30 18:38:26.895058 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.894660    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw99q\" (UniqueName: \"kubernetes.io/projected/9defd701-4377-47c0-bfaa-a298d0acc866-kube-api-access-zw99q\") pod \"iptables-alerter-gsknn\" (UID: \"9defd701-4377-47c0-bfaa-a298d0acc866\") " pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:38:26.895058 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.894716    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2l6s\" (UniqueName: \"kubernetes.io/projected/5b6f291a-fb97-4253-b3f1-39d0d482ca88-kube-api-access-w2l6s\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.895058 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.894837    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pgvf\" (UniqueName: \"kubernetes.io/projected/ce2cd81e-eece-495b-98d8-73fc05aef957-kube-api-access-7pgvf\") pod \"node-ca-5lhq5\" (UID: \"ce2cd81e-eece-495b-98d8-73fc05aef957\") " pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:38:26.895058 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.894853    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nspw\" (UniqueName: \"kubernetes.io/projected/38cac2ec-1eff-4426-a43b-ebe89a226412-kube-api-access-5nspw\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:38:26.895058 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.894868    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spnb6\" (UniqueName: \"kubernetes.io/projected/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-kube-api-access-spnb6\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:26.895058 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.894900    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtngp\" (UniqueName: \"kubernetes.io/projected/728e821c-4208-4b85-8b79-774997d92305-kube-api-access-gtngp\") pod \"ingress-canary-mhgzr\" (UID: \"728e821c-4208-4b85-8b79-774997d92305\") " pod="openshift-ingress-canary/ingress-canary-mhgzr"
Oct 30 18:38:26.895058 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.895037    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sknx\" (UniqueName: \"kubernetes.io/projected/f93cc289-68ea-4c64-9c47-0c16b1cf5f04-kube-api-access-6sknx\") pod \"network-check-target-g92j9\" (UID: \"f93cc289-68ea-4c64-9c47-0c16b1cf5f04\") " pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:26.895342 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.895199    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrsg8\" (UniqueName: \"kubernetes.io/projected/131610dd-0ebb-4f53-8339-e987b304d6f7-kube-api-access-vrsg8\") pod \"migrator-66df5d496b-hqvpc\" (UID: \"131610dd-0ebb-4f53-8339-e987b304d6f7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hqvpc"
Oct 30 18:38:26.895342 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.895301    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjjll\" (UniqueName: \"kubernetes.io/projected/146371eb-2e54-477f-a7ba-c13450f54828-kube-api-access-xjjll\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.904108 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904091    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-slash\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.904185 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904116    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.904185 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904159    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-modprobe-d\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.904255 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904183    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-multus-certs\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.904255 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904192    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-slash\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.904255 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904206    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-lib-modules\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.904255 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904211    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.904255 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904231    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-multus-certs\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.904255 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904235    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:26.904423 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904259    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-etc-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.904423 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904282    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-ovn\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.904423 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904305    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d197c973-f7cd-4dff-b621-71fc64897eeb-dbus\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:26.904423 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904315    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-modprobe-d\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.904423 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904330    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-run\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.904423 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:26.904327    4511 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:38:26.904423 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904308    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-lib-modules\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.904423 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904362    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-ovn\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.904423 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904363    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-etc-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.904423 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904397    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d197c973-f7cd-4dff-b621-71fc64897eeb-dbus\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:26.904423 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:26.904404    4511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs podName:8dccf064-473f-461c-b188-19495b90f62c nodeName:}" failed. No retries permitted until 2026-10-30 18:38:27.404386235 +0000 UTC m=+1.922071126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs") pod "network-metrics-daemon-s7gn8" (UID: "8dccf064-473f-461c-b188-19495b90f62c") : secret "metrics-daemon-secret" not found
Oct 30 18:38:26.904423 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904397    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-run\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.904423 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904426    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-systemd\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904444    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-netns\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904463    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-run-systemd\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904465    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-kubelet\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904499    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-crio-socket\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904504    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-netns\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904520    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-kubelet\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904526    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-cnibin\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904560    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d197c973-f7cd-4dff-b621-71fc64897eeb-kubelet-config\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904561    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-cnibin\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904573    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-crio-socket\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904576    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-trusted-ca-bundle\") pod \"periodic-gathering-pd6n6-bmftf\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") " pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904592    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904620    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d197c973-f7cd-4dff-b621-71fc64897eeb-kubelet-config\") pod \"global-pull-secret-syncer-bwbj7\" (UID: \"d197c973-f7cd-4dff-b621-71fc64897eeb\") " pod="kube-system/global-pull-secret-syncer-bwbj7"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:26.904674    4511 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904680    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysconfig\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904699    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-cni-multus\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904717    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-socket-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:26.904735    4511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls podName:ad954eaf-a685-4ac8-99f1-52f54994e109 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:27.40472505 +0000 UTC m=+1.922409956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls") pod "dns-default-s6drh" (UID: "ad954eaf-a685-4ac8-99f1-52f54994e109") : secret "dns-default-metrics-tls" not found
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904748    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-cni-netd\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904754    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-socket-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904771    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysctl-d\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904780    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-cni-netd\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904771    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-cni-multus\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904790    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysctl-conf\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904784    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysconfig\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904804    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-var-lib-kubelet\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904820    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-system-cni-dir\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904845    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysctl-d\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904869    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-system-cni-dir\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904871    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-var-lib-kubelet\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.904877 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904890    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-kubelet\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904913    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-var-lib-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904912    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-sysctl-conf\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904929    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-sys\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904945    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-kubelet\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904959    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-sys\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904972    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-host\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904983    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-var-lib-openvswitch\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.904995    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-systemd-units\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905006    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-host\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905019    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-log-socket\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905025    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-systemd-units\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905044    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-hosts-file\") pod \"node-resolver-k4r8f\" (UID: \"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3\") " pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905048    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-log-socket\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905058    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9defd701-4377-47c0-bfaa-a298d0acc866-host-slash\") pod \"iptables-alerter-gsknn\" (UID: \"9defd701-4377-47c0-bfaa-a298d0acc866\") " pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905073    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-socket-dir-parent\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905078    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1b3ccb2-2cc4-4291-a39f-e2edfff638f3-hosts-file\") pod \"node-resolver-k4r8f\" (UID: \"b1b3ccb2-2cc4-4291-a39f-e2edfff638f3\") " pod="openshift-dns/node-resolver-k4r8f"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905086    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-cni-bin\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905100    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-conf-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905104    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9defd701-4377-47c0-bfaa-a298d0acc866-host-slash\") pod \"iptables-alerter-gsknn\" (UID: \"9defd701-4377-47c0-bfaa-a298d0acc866\") " pod="openshift-network-operator/iptables-alerter-gsknn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905113    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-etc-kubernetes\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905152    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905156    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-socket-dir-parent\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905161    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-conf-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905165    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-cni-bin\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905173    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-etc-kubernetes\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905191    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-registration-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905213    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-device-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905229    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905250    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905262    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-device-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905264    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905287    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-kubernetes\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905300    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905273    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-registration-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905302    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:26.905321    4511 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905233    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905340    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905416    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-kubernetes\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:26.905418    4511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls podName:32718cc8-e0d1-40fb-bc79-5e3085c6ea3a nodeName:}" failed. No retries permitted until 2026-10-30 18:38:27.405403422 +0000 UTC m=+1.923088321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8jhgk" (UID: "32718cc8-e0d1-40fb-bc79-5e3085c6ea3a") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905433    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38cac2ec-1eff-4426-a43b-ebe89a226412-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lsgdd\" (UID: \"38cac2ec-1eff-4426-a43b-ebe89a226412\") " pod="openshift-multus/multus-additional-cni-plugins-lsgdd"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905438    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-cnibin\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905469    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-cnibin\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905478    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-system-cni-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905503    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-cni-bin\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905529    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905581    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905583    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-var-lib-cni-bin\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905583    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-system-cni-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905600    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce2cd81e-eece-495b-98d8-73fc05aef957-host\") pod \"node-ca-5lhq5\" (UID: \"ce2cd81e-eece-495b-98d8-73fc05aef957\") " pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905620    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-run-netns\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905636    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce2cd81e-eece-495b-98d8-73fc05aef957-host\") pod \"node-ca-5lhq5\" (UID: \"ce2cd81e-eece-495b-98d8-73fc05aef957\") " pod="openshift-image-registry/node-ca-5lhq5"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905636    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-node-log\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905661    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-cni-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905664    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-node-log\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905677    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-sys-fs\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905681    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a647480-88ca-47a2-9377-2841ce2e14f9-host-run-netns\") pod \"ovnkube-node-n5qjv\" (UID: \"2a647480-88ca-47a2-9377-2841ce2e14f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905690    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-systemd\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905706    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-k8s-cni-cncf-io\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905717    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-multus-cni-dir\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905722    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-hostroot\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905731    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbf4cda5-bce1-417d-94db-213431b7dd0e-sys-fs\") pod \"aws-ebs-csi-driver-node-2kkf5\" (UID: \"cbf4cda5-bce1-417d-94db-213431b7dd0e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kkf5"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905751    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-host-run-k8s-cni-cncf-io\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905767    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/146371eb-2e54-477f-a7ba-c13450f54828-etc-systemd\") pod \"tuned-f2b5l\" (UID: \"146371eb-2e54-477f-a7ba-c13450f54828\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b5l"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905771    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:26.906093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.905816    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b6f291a-fb97-4253-b3f1-39d0d482ca88-hostroot\") pod \"multus-jnnbn\" (UID: \"5b6f291a-fb97-4253-b3f1-39d0d482ca88\") " pod="openshift-multus/multus-jnnbn"
Oct 30 18:38:26.907901 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:26.906308    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:27.003731 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:27.003703    4511 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:38:27.373814 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:27.373787    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:27.408584 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:27.408558    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:27.408712 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:27.408594    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:27.408712 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:27.408699    4511 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:38:27.408838 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:27.408743    4511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls podName:ad954eaf-a685-4ac8-99f1-52f54994e109 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:28.40873165 +0000 UTC m=+2.926416542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls") pod "dns-default-s6drh" (UID: "ad954eaf-a685-4ac8-99f1-52f54994e109") : secret "dns-default-metrics-tls" not found
Oct 30 18:38:27.408838 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:27.408770    4511 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:38:27.408838 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:27.408796    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:27.408838 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:27.408826    4511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs podName:8dccf064-473f-461c-b188-19495b90f62c nodeName:}" failed. No retries permitted until 2026-10-30 18:38:28.408809312 +0000 UTC m=+2.926494206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs") pod "network-metrics-daemon-s7gn8" (UID: "8dccf064-473f-461c-b188-19495b90f62c") : secret "metrics-daemon-secret" not found
Oct 30 18:38:27.408974 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:27.408855    4511 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:38:27.408974 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:27.408898    4511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls podName:32718cc8-e0d1-40fb-bc79-5e3085c6ea3a nodeName:}" failed. No retries permitted until 2026-10-30 18:38:28.408887589 +0000 UTC m=+2.926572469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8jhgk" (UID: "32718cc8-e0d1-40fb-bc79-5e3085c6ea3a") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:38:28.058142 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:28.058110    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:28.331646 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:28.331537    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:28.417764 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:28.417731    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:28.417764 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:28.417776    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:28.417999 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:28.417822    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:28.417999 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:28.417896    4511 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:38:28.417999 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:28.417896    4511 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:38:28.417999 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:28.417962    4511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs podName:8dccf064-473f-461c-b188-19495b90f62c nodeName:}" failed. No retries permitted until 2026-10-30 18:38:30.417943073 +0000 UTC m=+4.935627964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs") pod "network-metrics-daemon-s7gn8" (UID: "8dccf064-473f-461c-b188-19495b90f62c") : secret "metrics-daemon-secret" not found
Oct 30 18:38:28.417999 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:28.417968    4511 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:38:28.418208 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:28.417979    4511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls podName:32718cc8-e0d1-40fb-bc79-5e3085c6ea3a nodeName:}" failed. No retries permitted until 2026-10-30 18:38:30.417970563 +0000 UTC m=+4.935655457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8jhgk" (UID: "32718cc8-e0d1-40fb-bc79-5e3085c6ea3a") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:38:28.418208 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:28.418049    4511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls podName:ad954eaf-a685-4ac8-99f1-52f54994e109 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:30.418031174 +0000 UTC m=+4.935716068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls") pod "dns-default-s6drh" (UID: "ad954eaf-a685-4ac8-99f1-52f54994e109") : secret "dns-default-metrics-tls" not found
Oct 30 18:38:28.712815 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:28.712729    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pd6n6-bmftf" event={"ID":"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9","Type":"ContainerStarted","Data":"fddd3af59e7ea85b455522b95351118dedcc5c6a16bbb871dd7b1fd21b605989"}
Oct 30 18:38:28.712979 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:28.712830    4511 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:38:30.434810 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:30.434775    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:30.435282 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:30.434839    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:30.435282 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:30.434879    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:30.435282 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:30.434927    4511 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:38:30.435282 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:30.434980    4511 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:38:30.435282 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:30.435006    4511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls podName:32718cc8-e0d1-40fb-bc79-5e3085c6ea3a nodeName:}" failed. No retries permitted until 2026-10-30 18:38:34.43499262 +0000 UTC m=+8.952677498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8jhgk" (UID: "32718cc8-e0d1-40fb-bc79-5e3085c6ea3a") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:38:30.435282 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:30.434928    4511 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:38:30.435282 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:30.435027    4511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls podName:ad954eaf-a685-4ac8-99f1-52f54994e109 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:34.435013272 +0000 UTC m=+8.952698165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls") pod "dns-default-s6drh" (UID: "ad954eaf-a685-4ac8-99f1-52f54994e109") : secret "dns-default-metrics-tls" not found
Oct 30 18:38:30.435282 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:30.435060    4511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs podName:8dccf064-473f-461c-b188-19495b90f62c nodeName:}" failed. No retries permitted until 2026-10-30 18:38:34.435046606 +0000 UTC m=+8.952731498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs") pod "network-metrics-daemon-s7gn8" (UID: "8dccf064-473f-461c-b188-19495b90f62c") : secret "metrics-daemon-secret" not found
Oct 30 18:38:30.719086 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:30.719052    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4z6kx" event={"ID":"a1fefd23-c919-4384-85c6-116295c18142","Type":"ContainerStarted","Data":"9f82affa44465fad04bb4177148be361a1577bde4ea7f039dd833fb8154f8be9"}
Oct 30 18:38:31.673879 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:31.673845    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k4r8f_b1b3ccb2-2cc4-4291-a39f-e2edfff638f3/dns-node-resolver/0.log"
Oct 30 18:38:32.474474 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:32.474447    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5lhq5_ce2cd81e-eece-495b-98d8-73fc05aef957/node-ca/0.log"
Oct 30 18:38:33.273346 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:33.273318    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5qjv"
Oct 30 18:38:33.474644 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:33.474612    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mhgzr_728e821c-4208-4b85-8b79-774997d92305/serve-healthcheck-canary/0.log"
Oct 30 18:38:33.874774 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:33.874744    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-hqvpc_131610dd-0ebb-4f53-8339-e987b304d6f7/migrator/0.log"
Oct 30 18:38:34.463541 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:34.463506    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:34.463956 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:34.463548    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:34.463956 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:34.463580    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:34.463956 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:34.463659    4511 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:38:34.463956 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:34.463736    4511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs podName:8dccf064-473f-461c-b188-19495b90f62c nodeName:}" failed. No retries permitted until 2026-10-30 18:38:42.463719589 +0000 UTC m=+16.981404481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs") pod "network-metrics-daemon-s7gn8" (UID: "8dccf064-473f-461c-b188-19495b90f62c") : secret "metrics-daemon-secret" not found
Oct 30 18:38:34.463956 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:34.463921    4511 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:38:34.466033 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:34.466015    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32718cc8-e0d1-40fb-bc79-5e3085c6ea3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jhgk\" (UID: \"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a\") " pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:34.466108 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:34.466093    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad954eaf-a685-4ac8-99f1-52f54994e109-metrics-tls\") pod \"dns-default-s6drh\" (UID: \"ad954eaf-a685-4ac8-99f1-52f54994e109\") " pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:34.499378 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:34.499349    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:34.500436 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:34.500417    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8jhgk"
Oct 30 18:38:34.927494 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:34.927460    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s6drh"]
Oct 30 18:38:34.934975 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:34.934944    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad954eaf_a685_4ac8_99f1_52f54994e109.slice/crio-4e24696293a4807205df27265ea261fef8c9b3ff03a42a5e16920bf3d5b7cf16 WatchSource:0}: Error finding container 4e24696293a4807205df27265ea261fef8c9b3ff03a42a5e16920bf3d5b7cf16: Status 404 returned error can't find the container with id 4e24696293a4807205df27265ea261fef8c9b3ff03a42a5e16920bf3d5b7cf16
Oct 30 18:38:34.976916 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:34.976892    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8jhgk"]
Oct 30 18:38:34.986377 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:34.986354    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32718cc8_e0d1_40fb_bc79_5e3085c6ea3a.slice/crio-f767f7bedf5b6229d32ca188b4103265e0365fc19754269e3a06b9b61cc006a7 WatchSource:0}: Error finding container f767f7bedf5b6229d32ca188b4103265e0365fc19754269e3a06b9b61cc006a7: Status 404 returned error can't find the container with id f767f7bedf5b6229d32ca188b4103265e0365fc19754269e3a06b9b61cc006a7
Oct 30 18:38:35.733141 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:35.733086    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8jhgk" event={"ID":"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a","Type":"ContainerStarted","Data":"2c1a856598da3507eeeae04557b5692dff06db3aa212d922ad2803b993f4f467"}
Oct 30 18:38:35.733660 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:35.733172    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8jhgk" event={"ID":"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a","Type":"ContainerStarted","Data":"f767f7bedf5b6229d32ca188b4103265e0365fc19754269e3a06b9b61cc006a7"}
Oct 30 18:38:35.734177 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:35.734152    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s6drh" event={"ID":"ad954eaf-a685-4ac8-99f1-52f54994e109","Type":"ContainerStarted","Data":"4e24696293a4807205df27265ea261fef8c9b3ff03a42a5e16920bf3d5b7cf16"}
Oct 30 18:38:36.741031 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:36.740989    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8jhgk" event={"ID":"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a","Type":"ContainerStarted","Data":"4381701a11fa3e3545a8349680666334bdae7a1ec95eb3bde3585f9d680115c6"}
Oct 30 18:38:37.746544 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:37.746052    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s6drh" event={"ID":"ad954eaf-a685-4ac8-99f1-52f54994e109","Type":"ContainerStarted","Data":"9edd22f544584c059292ba7a9c71a2748325412643abe6c9f1dcf2b296dca77b"}
Oct 30 18:38:37.746544 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:37.746111    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s6drh" event={"ID":"ad954eaf-a685-4ac8-99f1-52f54994e109","Type":"ContainerStarted","Data":"809d735dcb9401b4d8bf6ae13cb46ae42edd7b7eb84eaa6578954079529e1286"}
Oct 30 18:38:37.746544 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:37.746184    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:38.751383 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:38.751344    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8jhgk" event={"ID":"32718cc8-e0d1-40fb-bc79-5e3085c6ea3a","Type":"ContainerStarted","Data":"8610d04572224023a375b886b05d640e44d706e73d5a21d22ab3a4e09c639428"}
Oct 30 18:38:40.155622 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:40.155584    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:38:41.091378 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.091348    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pvfhh"]
Oct 30 18:38:41.093453 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.093436    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pvfhh"
Oct 30 18:38:41.115839 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.115814    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cd5a8461-de4f-4e73-bf84-96577e299b46-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-pvfhh\" (UID: \"cd5a8461-de4f-4e73-bf84-96577e299b46\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pvfhh"
Oct 30 18:38:41.146438 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.146403    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 18:38:41.190293 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.190267    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-qql85\""
Oct 30 18:38:41.192601 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.192579    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pvfhh"]
Oct 30 18:38:41.192994 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.192979    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d7858f76f-9jnk6"]
Oct 30 18:38:41.194933 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.194916    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.216532 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.216503    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0362ae67-1a48-42f8-9548-cc606aab2e02-trusted-ca\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.216629 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.216541    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb5pb\" (UniqueName: \"kubernetes.io/projected/0362ae67-1a48-42f8-9548-cc606aab2e02-kube-api-access-pb5pb\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.216629 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.216571    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0362ae67-1a48-42f8-9548-cc606aab2e02-image-registry-private-configuration\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.216719 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.216632    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0362ae67-1a48-42f8-9548-cc606aab2e02-registry-tls\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.216719 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.216692    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0362ae67-1a48-42f8-9548-cc606aab2e02-installation-pull-secrets\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.216844 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.216738    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cd5a8461-de4f-4e73-bf84-96577e299b46-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-pvfhh\" (UID: \"cd5a8461-de4f-4e73-bf84-96577e299b46\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pvfhh"
Oct 30 18:38:41.216844 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:41.216824    4511 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Oct 30 18:38:41.216913 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.216844    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0362ae67-1a48-42f8-9548-cc606aab2e02-ca-trust-extracted\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.216913 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:41.216864    4511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd5a8461-de4f-4e73-bf84-96577e299b46-tls-certificates podName:cd5a8461-de4f-4e73-bf84-96577e299b46 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:41.716851758 +0000 UTC m=+16.234536636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/cd5a8461-de4f-4e73-bf84-96577e299b46-tls-certificates") pod "prometheus-operator-admission-webhook-6489d8bf89-pvfhh" (UID: "cd5a8461-de4f-4e73-bf84-96577e299b46") : secret "prometheus-operator-admission-webhook-tls" not found
Oct 30 18:38:41.216913 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.216884    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0362ae67-1a48-42f8-9548-cc606aab2e02-registry-certificates\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.216913 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.216901    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0362ae67-1a48-42f8-9548-cc606aab2e02-bound-sa-token\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.300008 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.299978    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 18:38:41.317479 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.317447    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0362ae67-1a48-42f8-9548-cc606aab2e02-trusted-ca\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.317594 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.317489    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb5pb\" (UniqueName: \"kubernetes.io/projected/0362ae67-1a48-42f8-9548-cc606aab2e02-kube-api-access-pb5pb\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.317594 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.317520    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0362ae67-1a48-42f8-9548-cc606aab2e02-image-registry-private-configuration\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.317594 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.317546    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0362ae67-1a48-42f8-9548-cc606aab2e02-registry-tls\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.317594 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.317573    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0362ae67-1a48-42f8-9548-cc606aab2e02-installation-pull-secrets\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.317803 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.317739    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0362ae67-1a48-42f8-9548-cc606aab2e02-ca-trust-extracted\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.317803 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.317777    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0362ae67-1a48-42f8-9548-cc606aab2e02-registry-certificates\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.317902 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.317803    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0362ae67-1a48-42f8-9548-cc606aab2e02-bound-sa-token\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.318048 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.318025    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0362ae67-1a48-42f8-9548-cc606aab2e02-ca-trust-extracted\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.318507 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.318488    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0362ae67-1a48-42f8-9548-cc606aab2e02-trusted-ca\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.318613 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.318594    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0362ae67-1a48-42f8-9548-cc606aab2e02-registry-certificates\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.318826 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.318806    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 18:38:41.319011 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.318824    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x48k7\""
Oct 30 18:38:41.319076 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.319047    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 18:38:41.319226 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.319209    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 18:38:41.319849 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.319833    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d7858f76f-9jnk6"]
Oct 30 18:38:41.330726 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.330707    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0362ae67-1a48-42f8-9548-cc606aab2e02-image-registry-private-configuration\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.330832 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.330711    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0362ae67-1a48-42f8-9548-cc606aab2e02-installation-pull-secrets\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.330832 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.330818    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0362ae67-1a48-42f8-9548-cc606aab2e02-registry-tls\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.376201 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.376122    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb5pb\" (UniqueName: \"kubernetes.io/projected/0362ae67-1a48-42f8-9548-cc606aab2e02-kube-api-access-pb5pb\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.376384 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.376364    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0362ae67-1a48-42f8-9548-cc606aab2e02-bound-sa-token\") pod \"image-registry-5d7858f76f-9jnk6\" (UID: \"0362ae67-1a48-42f8-9548-cc606aab2e02\") " pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.504892 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.504859    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:41.720915 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.720843    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cd5a8461-de4f-4e73-bf84-96577e299b46-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-pvfhh\" (UID: \"cd5a8461-de4f-4e73-bf84-96577e299b46\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pvfhh"
Oct 30 18:38:41.721051 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:41.720961    4511 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Oct 30 18:38:41.721051 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:38:41.721022    4511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd5a8461-de4f-4e73-bf84-96577e299b46-tls-certificates podName:cd5a8461-de4f-4e73-bf84-96577e299b46 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:42.721008081 +0000 UTC m=+17.238692972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/cd5a8461-de4f-4e73-bf84-96577e299b46-tls-certificates") pod "prometheus-operator-admission-webhook-6489d8bf89-pvfhh" (UID: "cd5a8461-de4f-4e73-bf84-96577e299b46") : secret "prometheus-operator-admission-webhook-tls" not found
Oct 30 18:38:41.927756 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:41.927724    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d7858f76f-9jnk6"]
Oct 30 18:38:41.935764 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:41.935738    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0362ae67_1a48_42f8_9548_cc606aab2e02.slice/crio-82bd3631108b64baccf30eb846644fa8deaefc82344501541993537d8d355ca8 WatchSource:0}: Error finding container 82bd3631108b64baccf30eb846644fa8deaefc82344501541993537d8d355ca8: Status 404 returned error can't find the container with id 82bd3631108b64baccf30eb846644fa8deaefc82344501541993537d8d355ca8
Oct 30 18:38:42.528169 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:42.528118    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:42.530733 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:42.530703    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dccf064-473f-461c-b188-19495b90f62c-metrics-certs\") pod \"network-metrics-daemon-s7gn8\" (UID: \"8dccf064-473f-461c-b188-19495b90f62c\") " pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:42.596247 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:42.596214    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7gn8"
Oct 30 18:38:42.729236 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:42.729179    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cd5a8461-de4f-4e73-bf84-96577e299b46-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-pvfhh\" (UID: \"cd5a8461-de4f-4e73-bf84-96577e299b46\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pvfhh"
Oct 30 18:38:42.731752 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:42.731721    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cd5a8461-de4f-4e73-bf84-96577e299b46-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-pvfhh\" (UID: \"cd5a8461-de4f-4e73-bf84-96577e299b46\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pvfhh"
Oct 30 18:38:42.765415 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:42.765384    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6" event={"ID":"0362ae67-1a48-42f8-9548-cc606aab2e02","Type":"ContainerStarted","Data":"a53f8f05de01c0c7b656a6c2d39cba460cd6151833a143eb7c582e95a13810f6"}
Oct 30 18:38:42.765564 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:42.765429    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6" event={"ID":"0362ae67-1a48-42f8-9548-cc606aab2e02","Type":"ContainerStarted","Data":"82bd3631108b64baccf30eb846644fa8deaefc82344501541993537d8d355ca8"}
Oct 30 18:38:42.765564 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:42.765512    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:38:42.881436 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:42.881377    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6" podStartSLOduration=1.881360875 podStartE2EDuration="1.881360875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:42.880863519 +0000 UTC m=+17.398548418" watchObservedRunningTime="2026-10-30 18:38:42.881360875 +0000 UTC m=+17.399045775"
Oct 30 18:38:42.903039 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:42.903003    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pvfhh"
Oct 30 18:38:42.998050 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:42.998014    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s7gn8"]
Oct 30 18:38:43.005463 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:43.005438    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dccf064_473f_461c_b188_19495b90f62c.slice/crio-5a4f0a4a6c6b570cb231a736784d1b5c8981ebfd0040bc788dcdfe7da1abce69 WatchSource:0}: Error finding container 5a4f0a4a6c6b570cb231a736784d1b5c8981ebfd0040bc788dcdfe7da1abce69: Status 404 returned error can't find the container with id 5a4f0a4a6c6b570cb231a736784d1b5c8981ebfd0040bc788dcdfe7da1abce69
Oct 30 18:38:43.266901 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:43.266822    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pvfhh"]
Oct 30 18:38:43.274793 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:43.274753    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd5a8461_de4f_4e73_bf84_96577e299b46.slice/crio-64851c4778ed4c0e36cb9857be2149b952f25dbe8d0921cc60a8428816d01a1b WatchSource:0}: Error finding container 64851c4778ed4c0e36cb9857be2149b952f25dbe8d0921cc60a8428816d01a1b: Status 404 returned error can't find the container with id 64851c4778ed4c0e36cb9857be2149b952f25dbe8d0921cc60a8428816d01a1b
Oct 30 18:38:43.773931 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:43.773891    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pvfhh" event={"ID":"cd5a8461-de4f-4e73-bf84-96577e299b46","Type":"ContainerStarted","Data":"64851c4778ed4c0e36cb9857be2149b952f25dbe8d0921cc60a8428816d01a1b"}
Oct 30 18:38:43.774983 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:43.774961    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s7gn8" event={"ID":"8dccf064-473f-461c-b188-19495b90f62c","Type":"ContainerStarted","Data":"5a4f0a4a6c6b570cb231a736784d1b5c8981ebfd0040bc788dcdfe7da1abce69"}
Oct 30 18:38:44.781089 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:44.781006    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s7gn8" event={"ID":"8dccf064-473f-461c-b188-19495b90f62c","Type":"ContainerStarted","Data":"e96ceea18e99616eff13be20700d78878dab1a968e61233ee20253859d2c997d"}
Oct 30 18:38:44.781089 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:44.781068    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s7gn8" event={"ID":"8dccf064-473f-461c-b188-19495b90f62c","Type":"ContainerStarted","Data":"d99d5e61fd6f7f286f83b41e404c89e47197a3666c29fe9840fa56ed3d8b39d2"}
Oct 30 18:38:45.785104 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:45.785066    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pvfhh" event={"ID":"cd5a8461-de4f-4e73-bf84-96577e299b46","Type":"ContainerStarted","Data":"7cb8f286696301a546498926741fb1630950f18f94d00607d386d0be3315dfdb"}
Oct 30 18:38:45.895281 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:45.895229    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pvfhh" podStartSLOduration=3.215012356 podStartE2EDuration="4.895214679s" totalImagesPullingTime="1.680202323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:45.89511035 +0000 UTC m=+20.412795249" watchObservedRunningTime="2026-10-30 18:38:45.895214679 +0000 UTC m=+20.412899575"
Oct 30 18:38:46.788115 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:46.788078    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pvfhh"
Oct 30 18:38:46.792693 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:46.792667    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pvfhh"
Oct 30 18:38:47.753835 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:47.753811    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-s6drh"
Oct 30 18:38:54.333368 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.333339    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zpscf"]
Oct 30 18:38:54.336245 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.336222    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.386574 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.386544    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 18:38:54.407070 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.407047    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-root\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.407157 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.407075    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-sys\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.407157 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.407092    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-node-exporter-textfile\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.407157 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.407109    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-metrics-client-ca\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.407287 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.407272    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-node-exporter-wtmp\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.407321 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.407301    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btj9g\" (UniqueName: \"kubernetes.io/projected/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-kube-api-access-btj9g\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.407355 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.407319    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-node-exporter-accelerators-collector-config\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.407429 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.407410    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.407482 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.407467    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-node-exporter-tls\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.433774 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.433752    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 18:38:54.433895 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.433879    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w6qhz\""
Oct 30 18:38:54.433971 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.433921    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:38:54.434018 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.433975    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:38:54.434018 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.433940    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 18:38:54.434085 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.434075    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 18:38:54.508444 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.508409    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btj9g\" (UniqueName: \"kubernetes.io/projected/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-kube-api-access-btj9g\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.508444 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.508447    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-node-exporter-accelerators-collector-config\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.508658 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.508472    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.508658 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.508494    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-node-exporter-tls\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.508658 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.508525    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-root\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.508658 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.508550    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-sys\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.508658 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.508575    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-node-exporter-textfile\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.508658 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.508600    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-metrics-client-ca\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.508658 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.508633    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-node-exporter-wtmp\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.508658 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.508652    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-sys\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.509019 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.508692    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-root\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.509019 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.508767    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-node-exporter-wtmp\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.509019 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.508934    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-node-exporter-textfile\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.509109 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.509034    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-node-exporter-accelerators-collector-config\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.509188 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.509169    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-metrics-client-ca\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.510978 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.510958    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.511060 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.511045    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-node-exporter-tls\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.567340 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.567311    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btj9g\" (UniqueName: \"kubernetes.io/projected/f83aee63-bc05-4a59-8ca1-16abf2d6b7fe-kube-api-access-btj9g\") pod \"node-exporter-zpscf\" (UID: \"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe\") " pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.647573 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.647508    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zpscf"
Oct 30 18:38:54.658052 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:54.658028    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83aee63_bc05_4a59_8ca1_16abf2d6b7fe.slice/crio-220bc334524c12b0182c367bf584d84332e45308da3e1b37715ad2276d8171b9 WatchSource:0}: Error finding container 220bc334524c12b0182c367bf584d84332e45308da3e1b37715ad2276d8171b9: Status 404 returned error can't find the container with id 220bc334524c12b0182c367bf584d84332e45308da3e1b37715ad2276d8171b9
Oct 30 18:38:54.811912 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:54.811874    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zpscf" event={"ID":"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe","Type":"ContainerStarted","Data":"220bc334524c12b0182c367bf584d84332e45308da3e1b37715ad2276d8171b9"}
Oct 30 18:38:55.338012 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:55.337983    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dvpdt"
Oct 30 18:38:56.819527 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:56.819489    4511 generic.go:356] "Generic (PLEG): container finished" podID="f83aee63-bc05-4a59-8ca1-16abf2d6b7fe" containerID="0f6df48493db4f9518ecda965d0150f6da83ee25ede908f2cfe9469c394a4fc0" exitCode=0
Oct 30 18:38:56.820023 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:56.819541    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zpscf" event={"ID":"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe","Type":"ContainerDied","Data":"0f6df48493db4f9518ecda965d0150f6da83ee25ede908f2cfe9469c394a4fc0"}
Oct 30 18:38:57.710717 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:57.710691    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g92j9"
Oct 30 18:38:57.825097 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:57.825057    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zpscf" event={"ID":"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe","Type":"ContainerStarted","Data":"09a9593560ed79df468bc7ecd29cf88f638f4a22bd78b3419c5c47ba11ddacbe"}
Oct 30 18:38:57.825523 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:57.825115    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zpscf" event={"ID":"f83aee63-bc05-4a59-8ca1-16abf2d6b7fe","Type":"ContainerStarted","Data":"5bcab3eb839e798191b353cd84f4ff9435eab4f14b28ae33f74bff318919bf63"}
Oct 30 18:38:57.940094 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:57.940047    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zpscf" podStartSLOduration=2.434603089 podStartE2EDuration="3.940033218s" totalImagesPullingTime="1.431847864s" totalInitContainerRuntime="73.582265ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:57.939346751 +0000 UTC m=+32.457031650" watchObservedRunningTime="2026-10-30 18:38:57.940033218 +0000 UTC m=+32.457718114"
Oct 30 18:38:59.049261 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:59.049221    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-99qt8"]
Oct 30 18:38:59.080272 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:59.080244    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-99qt8"
Oct 30 18:38:59.106441 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:59.106413    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-99qt8"]
Oct 30 18:38:59.143929 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:59.143895    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5l22\" (UniqueName: \"kubernetes.io/projected/58ff6b78-bcbb-4759-bd03-3a0537510e42-kube-api-access-x5l22\") pod \"monitoring-plugin-77f4f76594-99qt8\" (UID: \"58ff6b78-bcbb-4759-bd03-3a0537510e42\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-99qt8"
Oct 30 18:38:59.144093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:59.143943    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/58ff6b78-bcbb-4759-bd03-3a0537510e42-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-99qt8\" (UID: \"58ff6b78-bcbb-4759-bd03-3a0537510e42\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-99qt8"
Oct 30 18:38:59.237545 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:59.237520    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 18:38:59.237724 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:59.237703    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-g2xhx\""
Oct 30 18:38:59.245067 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:59.245048    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5l22\" (UniqueName: \"kubernetes.io/projected/58ff6b78-bcbb-4759-bd03-3a0537510e42-kube-api-access-x5l22\") pod \"monitoring-plugin-77f4f76594-99qt8\" (UID: \"58ff6b78-bcbb-4759-bd03-3a0537510e42\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-99qt8"
Oct 30 18:38:59.245158 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:59.245094    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/58ff6b78-bcbb-4759-bd03-3a0537510e42-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-99qt8\" (UID: \"58ff6b78-bcbb-4759-bd03-3a0537510e42\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-99qt8"
Oct 30 18:38:59.247609 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:59.247587    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/58ff6b78-bcbb-4759-bd03-3a0537510e42-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-99qt8\" (UID: \"58ff6b78-bcbb-4759-bd03-3a0537510e42\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-99qt8"
Oct 30 18:38:59.307207 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:59.307117    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5l22\" (UniqueName: \"kubernetes.io/projected/58ff6b78-bcbb-4759-bd03-3a0537510e42-kube-api-access-x5l22\") pod \"monitoring-plugin-77f4f76594-99qt8\" (UID: \"58ff6b78-bcbb-4759-bd03-3a0537510e42\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-99qt8"
Oct 30 18:38:59.402976 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:59.402947    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-99qt8"
Oct 30 18:38:59.782551 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:59.782470    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-99qt8"]
Oct 30 18:38:59.789812 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:38:59.789777    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ff6b78_bcbb_4759_bd03_3a0537510e42.slice/crio-e978b027a621ed6d20fd4603923b97b98ce193a945c71c84dfd3dd6577ea1869 WatchSource:0}: Error finding container e978b027a621ed6d20fd4603923b97b98ce193a945c71c84dfd3dd6577ea1869: Status 404 returned error can't find the container with id e978b027a621ed6d20fd4603923b97b98ce193a945c71c84dfd3dd6577ea1869
Oct 30 18:38:59.833039 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:38:59.833011    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-99qt8" event={"ID":"58ff6b78-bcbb-4759-bd03-3a0537510e42","Type":"ContainerStarted","Data":"e978b027a621ed6d20fd4603923b97b98ce193a945c71c84dfd3dd6577ea1869"}
Oct 30 18:39:00.071328 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:00.071292    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-zr2j7"]
Oct 30 18:39:00.103366 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:00.103334    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-zr2j7"
Oct 30 18:39:00.130067 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:00.130043    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-zr2j7"]
Oct 30 18:39:00.150370 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:00.150351    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e09ad47-0f10-4b7d-a229-13390069f458-tmp\") pod \"downloads-6dd789c8b7-zr2j7\" (UID: \"4e09ad47-0f10-4b7d-a229-13390069f458\") " pod="openshift-console/downloads-6dd789c8b7-zr2j7"
Oct 30 18:39:00.150496 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:00.150388    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fff5\" (UniqueName: \"kubernetes.io/projected/4e09ad47-0f10-4b7d-a229-13390069f458-kube-api-access-9fff5\") pod \"downloads-6dd789c8b7-zr2j7\" (UID: \"4e09ad47-0f10-4b7d-a229-13390069f458\") " pod="openshift-console/downloads-6dd789c8b7-zr2j7"
Oct 30 18:39:00.254792 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:00.251743    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e09ad47-0f10-4b7d-a229-13390069f458-tmp\") pod \"downloads-6dd789c8b7-zr2j7\" (UID: \"4e09ad47-0f10-4b7d-a229-13390069f458\") " pod="openshift-console/downloads-6dd789c8b7-zr2j7"
Oct 30 18:39:00.254792 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:00.251809    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fff5\" (UniqueName: \"kubernetes.io/projected/4e09ad47-0f10-4b7d-a229-13390069f458-kube-api-access-9fff5\") pod \"downloads-6dd789c8b7-zr2j7\" (UID: \"4e09ad47-0f10-4b7d-a229-13390069f458\") " pod="openshift-console/downloads-6dd789c8b7-zr2j7"
Oct 30 18:39:00.255075 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:00.255053    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 18:39:00.255152 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:00.255066    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 18:39:00.255447 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:00.255432    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-fz4qs\""
Oct 30 18:39:00.255500 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:00.255481    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e09ad47-0f10-4b7d-a229-13390069f458-tmp\") pod \"downloads-6dd789c8b7-zr2j7\" (UID: \"4e09ad47-0f10-4b7d-a229-13390069f458\") " pod="openshift-console/downloads-6dd789c8b7-zr2j7"
Oct 30 18:39:00.311190 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:00.311161    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fff5\" (UniqueName: \"kubernetes.io/projected/4e09ad47-0f10-4b7d-a229-13390069f458-kube-api-access-9fff5\") pod \"downloads-6dd789c8b7-zr2j7\" (UID: \"4e09ad47-0f10-4b7d-a229-13390069f458\") " pod="openshift-console/downloads-6dd789c8b7-zr2j7"
Oct 30 18:39:00.427350 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:00.427276    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-zr2j7"
Oct 30 18:39:00.914421 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:00.914392    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-zr2j7"]
Oct 30 18:39:00.922796 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:39:00.922771    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e09ad47_0f10_4b7d_a229_13390069f458.slice/crio-4058da39e8600496548dafc89250c1cc59c2ddb654d21893a52adc5943af15c0 WatchSource:0}: Error finding container 4058da39e8600496548dafc89250c1cc59c2ddb654d21893a52adc5943af15c0: Status 404 returned error can't find the container with id 4058da39e8600496548dafc89250c1cc59c2ddb654d21893a52adc5943af15c0
Oct 30 18:39:01.841020 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:01.840984    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-zr2j7" event={"ID":"4e09ad47-0f10-4b7d-a229-13390069f458","Type":"ContainerStarted","Data":"4058da39e8600496548dafc89250c1cc59c2ddb654d21893a52adc5943af15c0"}
Oct 30 18:39:02.846805 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:02.846766    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-99qt8" event={"ID":"58ff6b78-bcbb-4759-bd03-3a0537510e42","Type":"ContainerStarted","Data":"44ac333ab44e9ccf5ea727efb01c004ea4690da669e8e88c958fec466a765582"}
Oct 30 18:39:02.847293 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:02.846929    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-99qt8"
Oct 30 18:39:02.852772 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:02.852730    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-99qt8"
Oct 30 18:39:02.958930 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:02.958875    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-99qt8" podStartSLOduration=1.693440126 podStartE2EDuration="3.95885929s" totalImagesPullingTime="2.265419164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:02.958057328 +0000 UTC m=+37.475742229" watchObservedRunningTime="2026-10-30 18:39:02.95885929 +0000 UTC m=+37.476544190"
Oct 30 18:39:03.779819 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:03.779792    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d7858f76f-9jnk6"
Oct 30 18:39:10.159615 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:39:10.159524    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:39:11.017509 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.017473    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fdd4486ff-z77hj"]
Oct 30 18:39:11.019830 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.019805    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.118690 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.118653    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 18:39:11.119306 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.119280    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-49bz6\""
Oct 30 18:39:11.119490 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.119325    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 18:39:11.119490 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.119385    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 18:39:11.119490 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.119403    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 18:39:11.119490 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.119438    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 18:39:11.120486 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.120420    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdd4486ff-z77hj"]
Oct 30 18:39:11.132330 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.132305    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-config\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.132444 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.132344    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp68k\" (UniqueName: \"kubernetes.io/projected/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-kube-api-access-lp68k\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.132444 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.132378    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-oauth-config\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.132555 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.132440    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-tmp\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.132555 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.132525    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-service-ca\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.132656 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.132577    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-serving-cert\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.132656 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.132609    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-oauth-serving-cert\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.233785 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.233748    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-serving-cert\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.234261 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.233795    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-oauth-serving-cert\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.234261 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.233851    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-config\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.234261 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.233880    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp68k\" (UniqueName: \"kubernetes.io/projected/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-kube-api-access-lp68k\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.234261 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.233912    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-oauth-config\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.234261 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.233939    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-tmp\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.234530 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.234260    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-service-ca\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.234530 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.234367    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-tmp\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.234794 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.234757    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-oauth-serving-cert\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.234872 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.234844    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-config\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.234973 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.234923    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-service-ca\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.236577 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.236557    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-serving-cert\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.237175 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.237159    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-oauth-config\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.292985 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.292907    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp68k\" (UniqueName: \"kubernetes.io/projected/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-kube-api-access-lp68k\") pod \"console-5fdd4486ff-z77hj\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") " pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.332505 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.332468    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:11.708572 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.708541    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdd4486ff-z77hj"]
Oct 30 18:39:11.744391 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:39:11.744342    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43b12ad_fe0c_4c12_8316_c1775a29a9ac.slice/crio-818836ec58fb8056ce69495daf53261cb30677c238a87480dd66f4e4ed0d4b27 WatchSource:0}: Error finding container 818836ec58fb8056ce69495daf53261cb30677c238a87480dd66f4e4ed0d4b27: Status 404 returned error can't find the container with id 818836ec58fb8056ce69495daf53261cb30677c238a87480dd66f4e4ed0d4b27
Oct 30 18:39:11.881270 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:11.881239    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdd4486ff-z77hj" event={"ID":"e43b12ad-fe0c-4c12-8316-c1775a29a9ac","Type":"ContainerStarted","Data":"818836ec58fb8056ce69495daf53261cb30677c238a87480dd66f4e4ed0d4b27"}
Oct 30 18:39:15.899441 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:15.899399    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdd4486ff-z77hj" event={"ID":"e43b12ad-fe0c-4c12-8316-c1775a29a9ac","Type":"ContainerStarted","Data":"35ae0f7e68af2674a6be415015532d386e539137772f8c28baf7ca8b689a6aad"}
Oct 30 18:39:16.014094 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:16.014038    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5fdd4486ff-z77hj" podStartSLOduration=2.725003751 podStartE2EDuration="6.014024912s" totalImagesPullingTime="3.289021161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:16.013871396 +0000 UTC m=+50.531556310" watchObservedRunningTime="2026-10-30 18:39:16.014024912 +0000 UTC m=+50.531709812"
Oct 30 18:39:17.913605 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:17.913569    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-zr2j7" event={"ID":"4e09ad47-0f10-4b7d-a229-13390069f458","Type":"ContainerStarted","Data":"791c3b0ddb200e339e3cd7c1360fb9f7b19e78033fe7e7aa1a0c522359b0b09d"}
Oct 30 18:39:17.914335 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:17.914312    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-zr2j7"
Oct 30 18:39:17.920511 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:17.920461    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-zr2j7"
Oct 30 18:39:18.121322 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.121254    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-zr2j7" podStartSLOduration=1.379455076 podStartE2EDuration="18.121240097s" totalImagesPullingTime="16.741785021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:18.073447586 +0000 UTC m=+52.591132485" watchObservedRunningTime="2026-10-30 18:39:18.121240097 +0000 UTC m=+52.638925017"
Oct 30 18:39:18.396880 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.396844    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-766767df9c-vjsnr"]
Oct 30 18:39:18.401215 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.401187    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.495225 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.495189    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc31efc4-3e87-4b47-8c76-b71118537f72-console-oauth-config\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.495471 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.495444    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc31efc4-3e87-4b47-8c76-b71118537f72-service-ca\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.495590 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.495486    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc31efc4-3e87-4b47-8c76-b71118537f72-console-config\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.495590 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.495526    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc31efc4-3e87-4b47-8c76-b71118537f72-console-serving-cert\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.495702 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.495636    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc31efc4-3e87-4b47-8c76-b71118537f72-trusted-ca-bundle\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.495702 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.495691    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc31efc4-3e87-4b47-8c76-b71118537f72-oauth-serving-cert\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.495798 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.495780    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2px49\" (UniqueName: \"kubernetes.io/projected/cc31efc4-3e87-4b47-8c76-b71118537f72-kube-api-access-2px49\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.495850 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.495815    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc31efc4-3e87-4b47-8c76-b71118537f72-tmp\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.500025 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.500002    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 18:39:18.501246 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.501223    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-766767df9c-vjsnr"]
Oct 30 18:39:18.596954 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.596909    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2px49\" (UniqueName: \"kubernetes.io/projected/cc31efc4-3e87-4b47-8c76-b71118537f72-kube-api-access-2px49\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.597171 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.596960    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc31efc4-3e87-4b47-8c76-b71118537f72-tmp\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.597171 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.597077    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc31efc4-3e87-4b47-8c76-b71118537f72-console-oauth-config\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.597171 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.597122    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc31efc4-3e87-4b47-8c76-b71118537f72-service-ca\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.597171 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.597169    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc31efc4-3e87-4b47-8c76-b71118537f72-console-config\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.597389 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.597209    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc31efc4-3e87-4b47-8c76-b71118537f72-console-serving-cert\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.597389 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.597251    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc31efc4-3e87-4b47-8c76-b71118537f72-trusted-ca-bundle\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.597389 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.597307    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc31efc4-3e87-4b47-8c76-b71118537f72-oauth-serving-cert\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.597540 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.597399    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc31efc4-3e87-4b47-8c76-b71118537f72-tmp\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.598434 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.598406    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc31efc4-3e87-4b47-8c76-b71118537f72-service-ca\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.598758 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.598737    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc31efc4-3e87-4b47-8c76-b71118537f72-console-config\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.598995 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.598898    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc31efc4-3e87-4b47-8c76-b71118537f72-trusted-ca-bundle\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.598995 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.598942    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc31efc4-3e87-4b47-8c76-b71118537f72-oauth-serving-cert\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.600536 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.600506    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc31efc4-3e87-4b47-8c76-b71118537f72-console-oauth-config\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.600643 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.600602    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc31efc4-3e87-4b47-8c76-b71118537f72-console-serving-cert\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.656510 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.656440    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2px49\" (UniqueName: \"kubernetes.io/projected/cc31efc4-3e87-4b47-8c76-b71118537f72-kube-api-access-2px49\") pod \"console-766767df9c-vjsnr\" (UID: \"cc31efc4-3e87-4b47-8c76-b71118537f72\") " pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:18.715399 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:18.715361    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:19.101012 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:19.100976    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-766767df9c-vjsnr"]
Oct 30 18:39:19.923797 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:19.923757    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-766767df9c-vjsnr" event={"ID":"cc31efc4-3e87-4b47-8c76-b71118537f72","Type":"ContainerStarted","Data":"30564e7dc195547f45ec4a87eae49f4414dad9a1091ac3f8345db908c0cedb63"}
Oct 30 18:39:19.923971 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:19.923820    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-766767df9c-vjsnr" event={"ID":"cc31efc4-3e87-4b47-8c76-b71118537f72","Type":"ContainerStarted","Data":"a4a48436e2ec0951b80448d48fd4ff1e47182c5ea60ae0d0e5622596ccccdd1a"}
Oct 30 18:39:20.038023 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:20.037973    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-766767df9c-vjsnr" podStartSLOduration=2.03795927 podStartE2EDuration="2.03795927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:20.037551113 +0000 UTC m=+54.555236039" watchObservedRunningTime="2026-10-30 18:39:20.03795927 +0000 UTC m=+54.555644188"
Oct 30 18:39:21.333119 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:21.333075    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:21.333581 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:21.333171    4511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:21.338759 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:21.338732    4511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:21.936921 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:21.936892    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:28.715685 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:28.715644    4511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:28.716213 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:28.715703    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:28.720345 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:28.720323    4511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:28.963074 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:28.963049    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-766767df9c-vjsnr"
Oct 30 18:39:29.106535 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:29.106509    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fdd4486ff-z77hj"]
Oct 30 18:39:40.155434 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:39:40.155392    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:39:47.458915 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:47.458884    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-pd6n6-bmftf"]
Oct 30 18:39:47.459504 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:47.459153    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-pd6n6-bmftf" podUID="6cc8f47a-4f4c-4adf-9b0b-668536b61bc9" containerName="insights-gathering" containerID="cri-o://fddd3af59e7ea85b455522b95351118dedcc5c6a16bbb871dd7b1fd21b605989" gracePeriod=30
Oct 30 18:39:47.772609 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:47.772580    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:39:47.924230 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:47.924200    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-service-ca-bundle\") pod \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") "
Oct 30 18:39:47.924379 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:47.924263    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-trusted-ca-bundle\") pod \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") "
Oct 30 18:39:47.924379 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:47.924287    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-kube-api-access-spnb6\" (UniqueName: \"kubernetes.io/projected/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-kube-api-access-spnb6\") pod \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") "
Oct 30 18:39:47.924379 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:47.924311    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-archives-path\") pod \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\" (UID: \"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9\") "
Oct 30 18:39:47.924661 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:47.924642    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-archives-path" pod "6cc8f47a-4f4c-4adf-9b0b-668536b61bc9" (UID: "6cc8f47a-4f4c-4adf-9b0b-668536b61bc9"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:39:47.924661 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:47.924647    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-trusted-ca-bundle" pod "6cc8f47a-4f4c-4adf-9b0b-668536b61bc9" (UID: "6cc8f47a-4f4c-4adf-9b0b-668536b61bc9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:47.924763 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:47.924629    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-service-ca-bundle" pod "6cc8f47a-4f4c-4adf-9b0b-668536b61bc9" (UID: "6cc8f47a-4f4c-4adf-9b0b-668536b61bc9"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:47.926689 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:47.926669    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-kube-api-access-spnb6" pod "6cc8f47a-4f4c-4adf-9b0b-668536b61bc9" (UID: "6cc8f47a-4f4c-4adf-9b0b-668536b61bc9"). InnerVolumeSpecName "kube-api-access-spnb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:39:48.019272 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:48.019207    4511 generic.go:356] "Generic (PLEG): container finished" podID="6cc8f47a-4f4c-4adf-9b0b-668536b61bc9" containerID="fddd3af59e7ea85b455522b95351118dedcc5c6a16bbb871dd7b1fd21b605989" exitCode=2
Oct 30 18:39:48.019272 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:48.019250    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pd6n6-bmftf" event={"ID":"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9","Type":"ContainerDied","Data":"fddd3af59e7ea85b455522b95351118dedcc5c6a16bbb871dd7b1fd21b605989"}
Oct 30 18:39:48.019272 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:48.019265    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pd6n6-bmftf"
Oct 30 18:39:48.019483 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:48.019286    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pd6n6-bmftf" event={"ID":"6cc8f47a-4f4c-4adf-9b0b-668536b61bc9","Type":"ContainerDied","Data":"a8cd336d75bdecabfafc7e272d16c8038312bbea957c6c242b945f80a4e9c24f"}
Oct 30 18:39:48.019483 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:48.019306    4511 scope.go:122] "RemoveContainer" containerID="fddd3af59e7ea85b455522b95351118dedcc5c6a16bbb871dd7b1fd21b605989"
Oct 30 18:39:48.025263 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:48.025241    4511 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-service-ca-bundle\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:48.025345 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:48.025270    4511 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-trusted-ca-bundle\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:48.025345 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:48.025284    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spnb6\" (UniqueName: \"kubernetes.io/projected/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-kube-api-access-spnb6\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:48.025345 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:48.025299    4511 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9-archives-path\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:48.027249 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:48.027233    4511 scope.go:122] "RemoveContainer" containerID="fddd3af59e7ea85b455522b95351118dedcc5c6a16bbb871dd7b1fd21b605989"
Oct 30 18:39:48.027483 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:39:48.027464    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fddd3af59e7ea85b455522b95351118dedcc5c6a16bbb871dd7b1fd21b605989\": container with ID starting with fddd3af59e7ea85b455522b95351118dedcc5c6a16bbb871dd7b1fd21b605989 not found: ID does not exist" containerID="fddd3af59e7ea85b455522b95351118dedcc5c6a16bbb871dd7b1fd21b605989"
Oct 30 18:39:48.027533 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:48.027490    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fddd3af59e7ea85b455522b95351118dedcc5c6a16bbb871dd7b1fd21b605989"} err="failed to get container status \"fddd3af59e7ea85b455522b95351118dedcc5c6a16bbb871dd7b1fd21b605989\": rpc error: code = NotFound desc = could not find container \"fddd3af59e7ea85b455522b95351118dedcc5c6a16bbb871dd7b1fd21b605989\": container with ID starting with fddd3af59e7ea85b455522b95351118dedcc5c6a16bbb871dd7b1fd21b605989 not found: ID does not exist"
Oct 30 18:39:48.175717 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:48.175689    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-pd6n6-bmftf"]
Oct 30 18:39:48.181506 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:48.181485    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-pd6n6-bmftf"]
Oct 30 18:39:49.655267 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:49.655230    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cc8f47a-4f4c-4adf-9b0b-668536b61bc9" path="/var/lib/kubelet/pods/6cc8f47a-4f4c-4adf-9b0b-668536b61bc9/volumes"
Oct 30 18:39:54.132304 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.132241    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5fdd4486ff-z77hj" podUID="e43b12ad-fe0c-4c12-8316-c1775a29a9ac" containerName="console" containerID="cri-o://35ae0f7e68af2674a6be415015532d386e539137772f8c28baf7ca8b689a6aad" gracePeriod=15
Oct 30 18:39:54.411742 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.411715    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:54.569696 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.569667    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-config\" (UniqueName: \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-config\") pod \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") "
Oct 30 18:39:54.569868 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.569718    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-oauth-serving-cert\") pod \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") "
Oct 30 18:39:54.569868 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.569743    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-tmp\") pod \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") "
Oct 30 18:39:54.569868 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.569774    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-kube-api-access-lp68k\" (UniqueName: \"kubernetes.io/projected/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-kube-api-access-lp68k\") pod \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") "
Oct 30 18:39:54.569868 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.569810    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-serving-cert\") pod \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") "
Oct 30 18:39:54.569868 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.569839    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-oauth-config\") pod \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") "
Oct 30 18:39:54.569868 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.569855    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-service-ca\") pod \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\" (UID: \"e43b12ad-fe0c-4c12-8316-c1775a29a9ac\") "
Oct 30 18:39:54.570202 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.570166    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-tmp" pod "e43b12ad-fe0c-4c12-8316-c1775a29a9ac" (UID: "e43b12ad-fe0c-4c12-8316-c1775a29a9ac"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:39:54.570256 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.570202    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-config" pod "e43b12ad-fe0c-4c12-8316-c1775a29a9ac" (UID: "e43b12ad-fe0c-4c12-8316-c1775a29a9ac"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:54.570256 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.570211    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-oauth-serving-cert" pod "e43b12ad-fe0c-4c12-8316-c1775a29a9ac" (UID: "e43b12ad-fe0c-4c12-8316-c1775a29a9ac"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:54.570459 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.570437    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-service-ca" pod "e43b12ad-fe0c-4c12-8316-c1775a29a9ac" (UID: "e43b12ad-fe0c-4c12-8316-c1775a29a9ac"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:54.572227 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.572192    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-serving-cert" pod "e43b12ad-fe0c-4c12-8316-c1775a29a9ac" (UID: "e43b12ad-fe0c-4c12-8316-c1775a29a9ac"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:54.572342 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.572252    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-kube-api-access-lp68k" pod "e43b12ad-fe0c-4c12-8316-c1775a29a9ac" (UID: "e43b12ad-fe0c-4c12-8316-c1775a29a9ac"). InnerVolumeSpecName "kube-api-access-lp68k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:39:54.572342 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.572269    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-oauth-config" pod "e43b12ad-fe0c-4c12-8316-c1775a29a9ac" (UID: "e43b12ad-fe0c-4c12-8316-c1775a29a9ac"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:54.670631 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.670575    4511 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:54.670631 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.670599    4511 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-oauth-serving-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:54.670631 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.670608    4511 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-tmp\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:54.670631 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.670618    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp68k\" (UniqueName: \"kubernetes.io/projected/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-kube-api-access-lp68k\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:54.670631 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.670627    4511 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-serving-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:54.670631 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.670637    4511 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-console-oauth-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:54.670881 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:54.670645    4511 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b12ad-fe0c-4c12-8316-c1775a29a9ac-service-ca\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:55.043198 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:55.043088    4511 generic.go:356] "Generic (PLEG): container finished" podID="e43b12ad-fe0c-4c12-8316-c1775a29a9ac" containerID="35ae0f7e68af2674a6be415015532d386e539137772f8c28baf7ca8b689a6aad" exitCode=0
Oct 30 18:39:55.043198 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:55.043162    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdd4486ff-z77hj"
Oct 30 18:39:55.043198 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:55.043162    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdd4486ff-z77hj" event={"ID":"e43b12ad-fe0c-4c12-8316-c1775a29a9ac","Type":"ContainerDied","Data":"35ae0f7e68af2674a6be415015532d386e539137772f8c28baf7ca8b689a6aad"}
Oct 30 18:39:55.043389 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:55.043220    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdd4486ff-z77hj" event={"ID":"e43b12ad-fe0c-4c12-8316-c1775a29a9ac","Type":"ContainerDied","Data":"818836ec58fb8056ce69495daf53261cb30677c238a87480dd66f4e4ed0d4b27"}
Oct 30 18:39:55.043389 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:55.043249    4511 scope.go:122] "RemoveContainer" containerID="35ae0f7e68af2674a6be415015532d386e539137772f8c28baf7ca8b689a6aad"
Oct 30 18:39:55.051154 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:55.051111    4511 scope.go:122] "RemoveContainer" containerID="35ae0f7e68af2674a6be415015532d386e539137772f8c28baf7ca8b689a6aad"
Oct 30 18:39:55.051389 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:39:55.051372    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ae0f7e68af2674a6be415015532d386e539137772f8c28baf7ca8b689a6aad\": container with ID starting with 35ae0f7e68af2674a6be415015532d386e539137772f8c28baf7ca8b689a6aad not found: ID does not exist" containerID="35ae0f7e68af2674a6be415015532d386e539137772f8c28baf7ca8b689a6aad"
Oct 30 18:39:55.051438 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:55.051401    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ae0f7e68af2674a6be415015532d386e539137772f8c28baf7ca8b689a6aad"} err="failed to get container status \"35ae0f7e68af2674a6be415015532d386e539137772f8c28baf7ca8b689a6aad\": rpc error: code = NotFound desc = could not find container \"35ae0f7e68af2674a6be415015532d386e539137772f8c28baf7ca8b689a6aad\": container with ID starting with 35ae0f7e68af2674a6be415015532d386e539137772f8c28baf7ca8b689a6aad not found: ID does not exist"
Oct 30 18:39:55.199999 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:55.199970    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fdd4486ff-z77hj"]
Oct 30 18:39:55.247555 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:55.247527    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fdd4486ff-z77hj"]
Oct 30 18:39:55.655664 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:39:55.655634    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e43b12ad-fe0c-4c12-8316-c1775a29a9ac" path="/var/lib/kubelet/pods/e43b12ad-fe0c-4c12-8316-c1775a29a9ac/volumes"
Oct 30 18:40:10.155188 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:40:10.155154    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:40:40.155727 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:40:40.155636    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:41:10.156052 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:41:10.156017    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:41:40.156103 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:41:40.156064    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:42:10.155960 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:42:10.155883    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:42:40.157305 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:42:40.157266    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:43:10.155511 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:43:10.155463    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:43:25.661190 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:25.661162    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 18:43:25.661676 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:25.661246    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 18:43:40.158354 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:43:40.158269    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:43:51.945987 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:51.945952    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg"]
Oct 30 18:43:51.946485 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:51.946235    4511 cpu_manager.go:414] "containerMap: removing container" podUID="e43b12ad-fe0c-4c12-8316-c1775a29a9ac" containerName="console"
Oct 30 18:43:51.946485 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:51.946247    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43b12ad-fe0c-4c12-8316-c1775a29a9ac" containerName="console"
Oct 30 18:43:51.946485 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:51.946260    4511 cpu_manager.go:414] "containerMap: removing container" podUID="6cc8f47a-4f4c-4adf-9b0b-668536b61bc9" containerName="insights-gathering"
Oct 30 18:43:51.946485 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:51.946265    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc8f47a-4f4c-4adf-9b0b-668536b61bc9" containerName="insights-gathering"
Oct 30 18:43:51.946485 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:51.946326    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43b12ad-fe0c-4c12-8316-c1775a29a9ac" containerName="console"
Oct 30 18:43:51.946485 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:51.946336    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc8f47a-4f4c-4adf-9b0b-668536b61bc9" containerName="insights-gathering"
Oct 30 18:43:51.948028 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:51.948012    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg"
Oct 30 18:43:52.046041 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.046011    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Oct 30 18:43:52.046041 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.046035    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Oct 30 18:43:52.046279 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.046069    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Oct 30 18:43:52.046279 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.046195    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Oct 30 18:43:52.046279 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.046217    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Oct 30 18:43:52.046279 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.046254    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-fvf6r\""
Oct 30 18:43:52.055285 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.055257    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/29c1a796-6512-46b5-81ce-d2531bb9ab8f-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-qw2gg\" (UID: \"29c1a796-6512-46b5-81ce-d2531bb9ab8f\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg"
Oct 30 18:43:52.055366 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.055289    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86zcc\" (UniqueName: \"kubernetes.io/projected/29c1a796-6512-46b5-81ce-d2531bb9ab8f-kube-api-access-86zcc\") pod \"keda-metrics-apiserver-86bd856cfb-qw2gg\" (UID: \"29c1a796-6512-46b5-81ce-d2531bb9ab8f\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg"
Oct 30 18:43:52.055366 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.055325    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/29c1a796-6512-46b5-81ce-d2531bb9ab8f-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-qw2gg\" (UID: \"29c1a796-6512-46b5-81ce-d2531bb9ab8f\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg"
Oct 30 18:43:52.087742 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.087720    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg"]
Oct 30 18:43:52.156649 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.156623    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/29c1a796-6512-46b5-81ce-d2531bb9ab8f-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-qw2gg\" (UID: \"29c1a796-6512-46b5-81ce-d2531bb9ab8f\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg"
Oct 30 18:43:52.156750 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.156655    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86zcc\" (UniqueName: \"kubernetes.io/projected/29c1a796-6512-46b5-81ce-d2531bb9ab8f-kube-api-access-86zcc\") pod \"keda-metrics-apiserver-86bd856cfb-qw2gg\" (UID: \"29c1a796-6512-46b5-81ce-d2531bb9ab8f\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg"
Oct 30 18:43:52.156750 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.156702    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/29c1a796-6512-46b5-81ce-d2531bb9ab8f-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-qw2gg\" (UID: \"29c1a796-6512-46b5-81ce-d2531bb9ab8f\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg"
Oct 30 18:43:52.157104 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.157081    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/29c1a796-6512-46b5-81ce-d2531bb9ab8f-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-qw2gg\" (UID: \"29c1a796-6512-46b5-81ce-d2531bb9ab8f\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg"
Oct 30 18:43:52.159397 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.159380    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/29c1a796-6512-46b5-81ce-d2531bb9ab8f-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-qw2gg\" (UID: \"29c1a796-6512-46b5-81ce-d2531bb9ab8f\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg"
Oct 30 18:43:52.215980 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.215934    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86zcc\" (UniqueName: \"kubernetes.io/projected/29c1a796-6512-46b5-81ce-d2531bb9ab8f-kube-api-access-86zcc\") pod \"keda-metrics-apiserver-86bd856cfb-qw2gg\" (UID: \"29c1a796-6512-46b5-81ce-d2531bb9ab8f\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg"
Oct 30 18:43:52.258830 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.258808    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg"
Oct 30 18:43:52.622575 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.622541    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg"]
Oct 30 18:43:52.630244 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:43:52.630218    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c1a796_6512_46b5_81ce_d2531bb9ab8f.slice/crio-1d5310c8a9aac674f5a21cb44d708626f9388deba5055050b0949053fe9b32b5 WatchSource:0}: Error finding container 1d5310c8a9aac674f5a21cb44d708626f9388deba5055050b0949053fe9b32b5: Status 404 returned error can't find the container with id 1d5310c8a9aac674f5a21cb44d708626f9388deba5055050b0949053fe9b32b5
Oct 30 18:43:52.631488 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.631472    4511 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:43:52.793660 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:52.793629    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg" event={"ID":"29c1a796-6512-46b5-81ce-d2531bb9ab8f","Type":"ContainerStarted","Data":"1d5310c8a9aac674f5a21cb44d708626f9388deba5055050b0949053fe9b32b5"}
Oct 30 18:43:57.811360 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:57.811317    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg" event={"ID":"29c1a796-6512-46b5-81ce-d2531bb9ab8f","Type":"ContainerStarted","Data":"6f900f63b2d83dc787b14d9f484987e2ed5d56afebb8264a3ba5e25acb713c8f"}
Oct 30 18:43:57.811761 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:57.811460    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg"
Oct 30 18:43:57.923433 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:43:57.923351    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg" podStartSLOduration=1.931152328 podStartE2EDuration="6.92333682s" totalImagesPullingTime="4.992184492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:57.92239189 +0000 UTC m=+332.440076789" watchObservedRunningTime="2026-10-30 18:43:57.92333682 +0000 UTC m=+332.441021719"
Oct 30 18:44:08.820971 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:44:08.820935    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-qw2gg"
Oct 30 18:44:10.156637 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:44:10.156593    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:44:40.159508 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:44:40.159463    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:45:10.155986 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:45:10.155907    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:45:13.493202 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.493171    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-844f887996-ktz9n"]
Oct 30 18:45:13.496253 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.496224    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-844f887996-ktz9n"
Oct 30 18:45:13.539552 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.539524    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-8kn2w"]
Oct 30 18:45:13.542270 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.542256    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-8kn2w"
Oct 30 18:45:13.586314 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.586290    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Oct 30 18:45:13.586441 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.586420    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 30 18:45:13.590251 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.590236    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 30 18:45:13.594100 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.594082    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-s7sj9\""
Oct 30 18:45:13.594990 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.594974    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-844f887996-ktz9n"]
Oct 30 18:45:13.620453 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.620431    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b05f7001-7e12-45dd-b6d6-98cd06ee1b3c-data\") pod \"seaweedfs-78bb9c6ff8-8kn2w\" (UID: \"b05f7001-7e12-45dd-b6d6-98cd06ee1b3c\") " pod="kserve/seaweedfs-78bb9c6ff8-8kn2w"
Oct 30 18:45:13.620563 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.620474    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx54k\" (UniqueName: \"kubernetes.io/projected/4a2f31bc-f2d1-430e-ab96-4ca62b231175-kube-api-access-lx54k\") pod \"kserve-controller-manager-844f887996-ktz9n\" (UID: \"4a2f31bc-f2d1-430e-ab96-4ca62b231175\") " pod="kserve/kserve-controller-manager-844f887996-ktz9n"
Oct 30 18:45:13.620612 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.620592    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a2f31bc-f2d1-430e-ab96-4ca62b231175-cert\") pod \"kserve-controller-manager-844f887996-ktz9n\" (UID: \"4a2f31bc-f2d1-430e-ab96-4ca62b231175\") " pod="kserve/kserve-controller-manager-844f887996-ktz9n"
Oct 30 18:45:13.620646 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.620635    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrltt\" (UniqueName: \"kubernetes.io/projected/b05f7001-7e12-45dd-b6d6-98cd06ee1b3c-kube-api-access-vrltt\") pod \"seaweedfs-78bb9c6ff8-8kn2w\" (UID: \"b05f7001-7e12-45dd-b6d6-98cd06ee1b3c\") " pod="kserve/seaweedfs-78bb9c6ff8-8kn2w"
Oct 30 18:45:13.635060 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.635032    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-8kn2w"]
Oct 30 18:45:13.646917 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.646894    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2bxz6\""
Oct 30 18:45:13.646917 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.646897    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Oct 30 18:45:13.721330 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.721311    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrltt\" (UniqueName: \"kubernetes.io/projected/b05f7001-7e12-45dd-b6d6-98cd06ee1b3c-kube-api-access-vrltt\") pod \"seaweedfs-78bb9c6ff8-8kn2w\" (UID: \"b05f7001-7e12-45dd-b6d6-98cd06ee1b3c\") " pod="kserve/seaweedfs-78bb9c6ff8-8kn2w"
Oct 30 18:45:13.721446 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.721349    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b05f7001-7e12-45dd-b6d6-98cd06ee1b3c-data\") pod \"seaweedfs-78bb9c6ff8-8kn2w\" (UID: \"b05f7001-7e12-45dd-b6d6-98cd06ee1b3c\") " pod="kserve/seaweedfs-78bb9c6ff8-8kn2w"
Oct 30 18:45:13.721446 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.721384    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx54k\" (UniqueName: \"kubernetes.io/projected/4a2f31bc-f2d1-430e-ab96-4ca62b231175-kube-api-access-lx54k\") pod \"kserve-controller-manager-844f887996-ktz9n\" (UID: \"4a2f31bc-f2d1-430e-ab96-4ca62b231175\") " pod="kserve/kserve-controller-manager-844f887996-ktz9n"
Oct 30 18:45:13.721561 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.721442    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a2f31bc-f2d1-430e-ab96-4ca62b231175-cert\") pod \"kserve-controller-manager-844f887996-ktz9n\" (UID: \"4a2f31bc-f2d1-430e-ab96-4ca62b231175\") " pod="kserve/kserve-controller-manager-844f887996-ktz9n"
Oct 30 18:45:13.721750 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.721731    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b05f7001-7e12-45dd-b6d6-98cd06ee1b3c-data\") pod \"seaweedfs-78bb9c6ff8-8kn2w\" (UID: \"b05f7001-7e12-45dd-b6d6-98cd06ee1b3c\") " pod="kserve/seaweedfs-78bb9c6ff8-8kn2w"
Oct 30 18:45:13.723902 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.723885    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a2f31bc-f2d1-430e-ab96-4ca62b231175-cert\") pod \"kserve-controller-manager-844f887996-ktz9n\" (UID: \"4a2f31bc-f2d1-430e-ab96-4ca62b231175\") " pod="kserve/kserve-controller-manager-844f887996-ktz9n"
Oct 30 18:45:13.780270 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.780215    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrltt\" (UniqueName: \"kubernetes.io/projected/b05f7001-7e12-45dd-b6d6-98cd06ee1b3c-kube-api-access-vrltt\") pod \"seaweedfs-78bb9c6ff8-8kn2w\" (UID: \"b05f7001-7e12-45dd-b6d6-98cd06ee1b3c\") " pod="kserve/seaweedfs-78bb9c6ff8-8kn2w"
Oct 30 18:45:13.780270 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.780261    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx54k\" (UniqueName: \"kubernetes.io/projected/4a2f31bc-f2d1-430e-ab96-4ca62b231175-kube-api-access-lx54k\") pod \"kserve-controller-manager-844f887996-ktz9n\" (UID: \"4a2f31bc-f2d1-430e-ab96-4ca62b231175\") " pod="kserve/kserve-controller-manager-844f887996-ktz9n"
Oct 30 18:45:13.808015 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.807994    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-844f887996-ktz9n"
Oct 30 18:45:13.853095 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:13.853065    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-8kn2w"
Oct 30 18:45:14.175815 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:14.175783    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-844f887996-ktz9n"]
Oct 30 18:45:14.222816 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:45:14.222786    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2f31bc_f2d1_430e_ab96_4ca62b231175.slice/crio-55d794ed4d76807baefe280ab364d704bfcd92ba409f824b13762fb971570fbd WatchSource:0}: Error finding container 55d794ed4d76807baefe280ab364d704bfcd92ba409f824b13762fb971570fbd: Status 404 returned error can't find the container with id 55d794ed4d76807baefe280ab364d704bfcd92ba409f824b13762fb971570fbd
Oct 30 18:45:14.226912 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:14.226893    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-8kn2w"]
Oct 30 18:45:14.233652 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:45:14.233631    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05f7001_7e12_45dd_b6d6_98cd06ee1b3c.slice/crio-6135164a946b47024c06f5be4e512f338d0a607af7eb75b386b1bd99d1ac65a3 WatchSource:0}: Error finding container 6135164a946b47024c06f5be4e512f338d0a607af7eb75b386b1bd99d1ac65a3: Status 404 returned error can't find the container with id 6135164a946b47024c06f5be4e512f338d0a607af7eb75b386b1bd99d1ac65a3
Oct 30 18:45:15.053571 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:15.053528    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-844f887996-ktz9n" event={"ID":"4a2f31bc-f2d1-430e-ab96-4ca62b231175","Type":"ContainerStarted","Data":"55d794ed4d76807baefe280ab364d704bfcd92ba409f824b13762fb971570fbd"}
Oct 30 18:45:15.054571 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:15.054547    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-8kn2w" event={"ID":"b05f7001-7e12-45dd-b6d6-98cd06ee1b3c","Type":"ContainerStarted","Data":"6135164a946b47024c06f5be4e512f338d0a607af7eb75b386b1bd99d1ac65a3"}
Oct 30 18:45:20.074567 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:20.074530    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-8kn2w" event={"ID":"b05f7001-7e12-45dd-b6d6-98cd06ee1b3c","Type":"ContainerStarted","Data":"41477d3234c905040c77df5f08de5c5546230fc024ef58e8b29919b390cc2d6d"}
Oct 30 18:45:20.074963 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:20.074639    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-78bb9c6ff8-8kn2w"
Oct 30 18:45:20.186064 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:20.186010    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-78bb9c6ff8-8kn2w" podStartSLOduration=2.232255193 podStartE2EDuration="7.185997307s" totalImagesPullingTime="4.953742114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:20.185107259 +0000 UTC m=+414.702792161" watchObservedRunningTime="2026-10-30 18:45:20.185997307 +0000 UTC m=+414.703682204"
Oct 30 18:45:21.083771 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:21.083730    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-844f887996-ktz9n" event={"ID":"4a2f31bc-f2d1-430e-ab96-4ca62b231175","Type":"ContainerStarted","Data":"f6d9fb5dfc999bffb845a6d1f8e64cf01c507b18c9edd4cec53e58ecb6977b3c"}
Oct 30 18:45:21.084247 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:21.083838    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-844f887996-ktz9n"
Oct 30 18:45:26.085906 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:26.085878    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-78bb9c6ff8-8kn2w"
Oct 30 18:45:26.200049 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:26.199994    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/kserve-controller-manager-844f887996-ktz9n" podStartSLOduration=6.763294845 podStartE2EDuration="13.199981679s" totalImagesPullingTime="6.436686834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:21.196559268 +0000 UTC m=+415.714244167" watchObservedRunningTime="2026-10-30 18:45:26.199981679 +0000 UTC m=+420.717666578"
Oct 30 18:45:40.158467 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:45:40.158416    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:45:52.092624 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:45:52.092593    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-844f887996-ktz9n"
Oct 30 18:46:08.948320 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:08.948286    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-844f887996-ktz9n"]
Oct 30 18:46:08.948714 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:08.948485    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="kserve/kserve-controller-manager-844f887996-ktz9n" podUID="4a2f31bc-f2d1-430e-ab96-4ca62b231175" containerName="manager" containerID="cri-o://f6d9fb5dfc999bffb845a6d1f8e64cf01c507b18c9edd4cec53e58ecb6977b3c" gracePeriod=10
Oct 30 18:46:08.970570 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:08.970548    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-844f887996-7kcmx"]
Oct 30 18:46:08.973403 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:08.973386    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-844f887996-7kcmx"
Oct 30 18:46:09.052757 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.052729    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-844f887996-7kcmx"]
Oct 30 18:46:09.111214 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.111185    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/250d515c-2d5a-4bda-8ee9-efb06c9bb900-cert\") pod \"kserve-controller-manager-844f887996-7kcmx\" (UID: \"250d515c-2d5a-4bda-8ee9-efb06c9bb900\") " pod="kserve/kserve-controller-manager-844f887996-7kcmx"
Oct 30 18:46:09.111375 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.111222    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlxhz\" (UniqueName: \"kubernetes.io/projected/250d515c-2d5a-4bda-8ee9-efb06c9bb900-kube-api-access-xlxhz\") pod \"kserve-controller-manager-844f887996-7kcmx\" (UID: \"250d515c-2d5a-4bda-8ee9-efb06c9bb900\") " pod="kserve/kserve-controller-manager-844f887996-7kcmx"
Oct 30 18:46:09.190598 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.190575    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-844f887996-ktz9n"
Oct 30 18:46:09.212556 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.212509    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/250d515c-2d5a-4bda-8ee9-efb06c9bb900-cert\") pod \"kserve-controller-manager-844f887996-7kcmx\" (UID: \"250d515c-2d5a-4bda-8ee9-efb06c9bb900\") " pod="kserve/kserve-controller-manager-844f887996-7kcmx"
Oct 30 18:46:09.212556 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.212537    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlxhz\" (UniqueName: \"kubernetes.io/projected/250d515c-2d5a-4bda-8ee9-efb06c9bb900-kube-api-access-xlxhz\") pod \"kserve-controller-manager-844f887996-7kcmx\" (UID: \"250d515c-2d5a-4bda-8ee9-efb06c9bb900\") " pod="kserve/kserve-controller-manager-844f887996-7kcmx"
Oct 30 18:46:09.214835 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.214813    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/250d515c-2d5a-4bda-8ee9-efb06c9bb900-cert\") pod \"kserve-controller-manager-844f887996-7kcmx\" (UID: \"250d515c-2d5a-4bda-8ee9-efb06c9bb900\") " pod="kserve/kserve-controller-manager-844f887996-7kcmx"
Oct 30 18:46:09.245313 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.245290    4511 generic.go:356] "Generic (PLEG): container finished" podID="4a2f31bc-f2d1-430e-ab96-4ca62b231175" containerID="f6d9fb5dfc999bffb845a6d1f8e64cf01c507b18c9edd4cec53e58ecb6977b3c" exitCode=0
Oct 30 18:46:09.245427 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.245328    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-844f887996-ktz9n" event={"ID":"4a2f31bc-f2d1-430e-ab96-4ca62b231175","Type":"ContainerDied","Data":"f6d9fb5dfc999bffb845a6d1f8e64cf01c507b18c9edd4cec53e58ecb6977b3c"}
Oct 30 18:46:09.245427 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.245337    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-844f887996-ktz9n"
Oct 30 18:46:09.245427 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.245367    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-844f887996-ktz9n" event={"ID":"4a2f31bc-f2d1-430e-ab96-4ca62b231175","Type":"ContainerDied","Data":"55d794ed4d76807baefe280ab364d704bfcd92ba409f824b13762fb971570fbd"}
Oct 30 18:46:09.245427 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.245387    4511 scope.go:122] "RemoveContainer" containerID="f6d9fb5dfc999bffb845a6d1f8e64cf01c507b18c9edd4cec53e58ecb6977b3c"
Oct 30 18:46:09.252813 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.252798    4511 scope.go:122] "RemoveContainer" containerID="f6d9fb5dfc999bffb845a6d1f8e64cf01c507b18c9edd4cec53e58ecb6977b3c"
Oct 30 18:46:09.253071 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:46:09.253053    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d9fb5dfc999bffb845a6d1f8e64cf01c507b18c9edd4cec53e58ecb6977b3c\": container with ID starting with f6d9fb5dfc999bffb845a6d1f8e64cf01c507b18c9edd4cec53e58ecb6977b3c not found: ID does not exist" containerID="f6d9fb5dfc999bffb845a6d1f8e64cf01c507b18c9edd4cec53e58ecb6977b3c"
Oct 30 18:46:09.253120 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.253082    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d9fb5dfc999bffb845a6d1f8e64cf01c507b18c9edd4cec53e58ecb6977b3c"} err="failed to get container status \"f6d9fb5dfc999bffb845a6d1f8e64cf01c507b18c9edd4cec53e58ecb6977b3c\": rpc error: code = NotFound desc = could not find container \"f6d9fb5dfc999bffb845a6d1f8e64cf01c507b18c9edd4cec53e58ecb6977b3c\": container with ID starting with f6d9fb5dfc999bffb845a6d1f8e64cf01c507b18c9edd4cec53e58ecb6977b3c not found: ID does not exist"
Oct 30 18:46:09.272623 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.272600    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlxhz\" (UniqueName: \"kubernetes.io/projected/250d515c-2d5a-4bda-8ee9-efb06c9bb900-kube-api-access-xlxhz\") pod \"kserve-controller-manager-844f887996-7kcmx\" (UID: \"250d515c-2d5a-4bda-8ee9-efb06c9bb900\") " pod="kserve/kserve-controller-manager-844f887996-7kcmx"
Oct 30 18:46:09.312979 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.312952    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a2f31bc-f2d1-430e-ab96-4ca62b231175-cert\" (UniqueName: \"kubernetes.io/secret/4a2f31bc-f2d1-430e-ab96-4ca62b231175-cert\") pod \"4a2f31bc-f2d1-430e-ab96-4ca62b231175\" (UID: \"4a2f31bc-f2d1-430e-ab96-4ca62b231175\") "
Oct 30 18:46:09.313092 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.313047    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2f31bc-f2d1-430e-ab96-4ca62b231175-kube-api-access-lx54k\" (UniqueName: \"kubernetes.io/projected/4a2f31bc-f2d1-430e-ab96-4ca62b231175-kube-api-access-lx54k\") pod \"4a2f31bc-f2d1-430e-ab96-4ca62b231175\" (UID: \"4a2f31bc-f2d1-430e-ab96-4ca62b231175\") "
Oct 30 18:46:09.315101 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.315073    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2f31bc-f2d1-430e-ab96-4ca62b231175-cert" pod "4a2f31bc-f2d1-430e-ab96-4ca62b231175" (UID: "4a2f31bc-f2d1-430e-ab96-4ca62b231175"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:46:09.315236 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.315210    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2f31bc-f2d1-430e-ab96-4ca62b231175-kube-api-access-lx54k" pod "4a2f31bc-f2d1-430e-ab96-4ca62b231175" (UID: "4a2f31bc-f2d1-430e-ab96-4ca62b231175"). InnerVolumeSpecName "kube-api-access-lx54k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:46:09.333478 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.333453    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-844f887996-7kcmx"
Oct 30 18:46:09.414630 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.414603    4511 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a2f31bc-f2d1-430e-ab96-4ca62b231175-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:09.414630 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.414627    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx54k\" (UniqueName: \"kubernetes.io/projected/4a2f31bc-f2d1-430e-ab96-4ca62b231175-kube-api-access-lx54k\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:09.660228 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.660199    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-844f887996-ktz9n"]
Oct 30 18:46:09.666688 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.666657    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-844f887996-ktz9n"]
Oct 30 18:46:09.700596 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:09.700575    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-844f887996-7kcmx"]
Oct 30 18:46:09.748332 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:46:09.748305    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250d515c_2d5a_4bda_8ee9_efb06c9bb900.slice/crio-94494f362906bc87841314ed9b4f618adbd1332ef0be2f828e08145d13a6ab1c WatchSource:0}: Error finding container 94494f362906bc87841314ed9b4f618adbd1332ef0be2f828e08145d13a6ab1c: Status 404 returned error can't find the container with id 94494f362906bc87841314ed9b4f618adbd1332ef0be2f828e08145d13a6ab1c
Oct 30 18:46:10.156411 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:46:10.156375    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:46:10.249655 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:10.249578    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-844f887996-7kcmx" event={"ID":"250d515c-2d5a-4bda-8ee9-efb06c9bb900","Type":"ContainerStarted","Data":"94494f362906bc87841314ed9b4f618adbd1332ef0be2f828e08145d13a6ab1c"}
Oct 30 18:46:11.255366 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:11.255329    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-844f887996-7kcmx" event={"ID":"250d515c-2d5a-4bda-8ee9-efb06c9bb900","Type":"ContainerStarted","Data":"af05e11fb5a7b248093e7648db10e2ea7bc67f39d8be2496e9f8c58ae8a6182e"}
Oct 30 18:46:11.255744 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:11.255476    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-844f887996-7kcmx"
Oct 30 18:46:11.367333 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:11.367287    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/kserve-controller-manager-844f887996-7kcmx" podStartSLOduration=2.473327546 podStartE2EDuration="3.367274661s" totalImagesPullingTime="893.947115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:11.367080177 +0000 UTC m=+465.884765076" watchObservedRunningTime="2026-10-30 18:46:11.367274661 +0000 UTC m=+465.884959560"
Oct 30 18:46:11.655869 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:11.655838    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a2f31bc-f2d1-430e-ab96-4ca62b231175" path="/var/lib/kubelet/pods/4a2f31bc-f2d1-430e-ab96-4ca62b231175/volumes"
Oct 30 18:46:40.156190 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:46:40.156097    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:46:42.265213 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:42.265171    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-844f887996-7kcmx"
Oct 30 18:46:44.264526 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.264491    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-56ddc4c448-lbktr"]
Oct 30 18:46:44.264974 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.264871    4511 cpu_manager.go:414] "containerMap: removing container" podUID="4a2f31bc-f2d1-430e-ab96-4ca62b231175" containerName="manager"
Oct 30 18:46:44.264974 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.264888    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2f31bc-f2d1-430e-ab96-4ca62b231175" containerName="manager"
Oct 30 18:46:44.265091 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.264977    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2f31bc-f2d1-430e-ab96-4ca62b231175" containerName="manager"
Oct 30 18:46:44.267647 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.267627    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-lbktr"
Oct 30 18:46:44.314859 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.314832    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-nv4z9"]
Oct 30 18:46:44.317608 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.317582    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-nv4z9"
Oct 30 18:46:44.343756 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.343733    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b38b02-20af-4975-bf9a-54c6a31e8b64-tls-certs\") pod \"model-serving-api-56ddc4c448-lbktr\" (UID: \"58b38b02-20af-4975-bf9a-54c6a31e8b64\") " pod="kserve/model-serving-api-56ddc4c448-lbktr"
Oct 30 18:46:44.343849 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.343790    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7qkp\" (UniqueName: \"kubernetes.io/projected/d634ca90-4998-45ba-a69d-276ff6ceae58-kube-api-access-j7qkp\") pod \"odh-model-controller-7f9f8cdc65-nv4z9\" (UID: \"d634ca90-4998-45ba-a69d-276ff6ceae58\") " pod="kserve/odh-model-controller-7f9f8cdc65-nv4z9"
Oct 30 18:46:44.343892 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.343848    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d634ca90-4998-45ba-a69d-276ff6ceae58-cert\") pod \"odh-model-controller-7f9f8cdc65-nv4z9\" (UID: \"d634ca90-4998-45ba-a69d-276ff6ceae58\") " pod="kserve/odh-model-controller-7f9f8cdc65-nv4z9"
Oct 30 18:46:44.343892 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.343866    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d634ca90-4998-45ba-a69d-276ff6ceae58-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-nv4z9\" (UID: \"d634ca90-4998-45ba-a69d-276ff6ceae58\") " pod="kserve/odh-model-controller-7f9f8cdc65-nv4z9"
Oct 30 18:46:44.343955 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.343917    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dkq7\" (UniqueName: \"kubernetes.io/projected/58b38b02-20af-4975-bf9a-54c6a31e8b64-kube-api-access-6dkq7\") pod \"model-serving-api-56ddc4c448-lbktr\" (UID: \"58b38b02-20af-4975-bf9a-54c6a31e8b64\") " pod="kserve/model-serving-api-56ddc4c448-lbktr"
Oct 30 18:46:44.361357 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.361340    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-t5cdr\""
Oct 30 18:46:44.361429 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.361347    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Oct 30 18:46:44.365120 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.365102    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-lbktr"]
Oct 30 18:46:44.367142 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.367110    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Oct 30 18:46:44.408569 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.408543    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Oct 30 18:46:44.408679 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.408659    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-7qx7w\""
Oct 30 18:46:44.409744 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.409727    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-nv4z9"]
Oct 30 18:46:44.445266 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.445243    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7qkp\" (UniqueName: \"kubernetes.io/projected/d634ca90-4998-45ba-a69d-276ff6ceae58-kube-api-access-j7qkp\") pod \"odh-model-controller-7f9f8cdc65-nv4z9\" (UID: \"d634ca90-4998-45ba-a69d-276ff6ceae58\") " pod="kserve/odh-model-controller-7f9f8cdc65-nv4z9"
Oct 30 18:46:44.445362 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.445274    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d634ca90-4998-45ba-a69d-276ff6ceae58-cert\") pod \"odh-model-controller-7f9f8cdc65-nv4z9\" (UID: \"d634ca90-4998-45ba-a69d-276ff6ceae58\") " pod="kserve/odh-model-controller-7f9f8cdc65-nv4z9"
Oct 30 18:46:44.445362 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.445294    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d634ca90-4998-45ba-a69d-276ff6ceae58-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-nv4z9\" (UID: \"d634ca90-4998-45ba-a69d-276ff6ceae58\") " pod="kserve/odh-model-controller-7f9f8cdc65-nv4z9"
Oct 30 18:46:44.445362 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.445351    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dkq7\" (UniqueName: \"kubernetes.io/projected/58b38b02-20af-4975-bf9a-54c6a31e8b64-kube-api-access-6dkq7\") pod \"model-serving-api-56ddc4c448-lbktr\" (UID: \"58b38b02-20af-4975-bf9a-54c6a31e8b64\") " pod="kserve/model-serving-api-56ddc4c448-lbktr"
Oct 30 18:46:44.445516 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.445393    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b38b02-20af-4975-bf9a-54c6a31e8b64-tls-certs\") pod \"model-serving-api-56ddc4c448-lbktr\" (UID: \"58b38b02-20af-4975-bf9a-54c6a31e8b64\") " pod="kserve/model-serving-api-56ddc4c448-lbktr"
Oct 30 18:46:44.448727 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.448698    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d634ca90-4998-45ba-a69d-276ff6ceae58-cert\") pod \"odh-model-controller-7f9f8cdc65-nv4z9\" (UID: \"d634ca90-4998-45ba-a69d-276ff6ceae58\") " pod="kserve/odh-model-controller-7f9f8cdc65-nv4z9"
Oct 30 18:46:44.452746 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.452724    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b38b02-20af-4975-bf9a-54c6a31e8b64-tls-certs\") pod \"model-serving-api-56ddc4c448-lbktr\" (UID: \"58b38b02-20af-4975-bf9a-54c6a31e8b64\") " pod="kserve/model-serving-api-56ddc4c448-lbktr"
Oct 30 18:46:44.452826 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.452805    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d634ca90-4998-45ba-a69d-276ff6ceae58-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-nv4z9\" (UID: \"d634ca90-4998-45ba-a69d-276ff6ceae58\") " pod="kserve/odh-model-controller-7f9f8cdc65-nv4z9"
Oct 30 18:46:44.503899 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.503874    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7qkp\" (UniqueName: \"kubernetes.io/projected/d634ca90-4998-45ba-a69d-276ff6ceae58-kube-api-access-j7qkp\") pod \"odh-model-controller-7f9f8cdc65-nv4z9\" (UID: \"d634ca90-4998-45ba-a69d-276ff6ceae58\") " pod="kserve/odh-model-controller-7f9f8cdc65-nv4z9"
Oct 30 18:46:44.504117 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.504095    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dkq7\" (UniqueName: \"kubernetes.io/projected/58b38b02-20af-4975-bf9a-54c6a31e8b64-kube-api-access-6dkq7\") pod \"model-serving-api-56ddc4c448-lbktr\" (UID: \"58b38b02-20af-4975-bf9a-54c6a31e8b64\") " pod="kserve/model-serving-api-56ddc4c448-lbktr"
Oct 30 18:46:44.579378 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.579360    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-lbktr"
Oct 30 18:46:44.630393 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.630362    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-nv4z9"
Oct 30 18:46:44.945004 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:44.944935    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-lbktr"]
Oct 30 18:46:44.997180 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:46:44.997154    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b38b02_20af_4975_bf9a_54c6a31e8b64.slice/crio-8d846f7c149a36f54b1a266e68623cf1e4fe47ff589deb6bba7efd877cbf00e8 WatchSource:0}: Error finding container 8d846f7c149a36f54b1a266e68623cf1e4fe47ff589deb6bba7efd877cbf00e8: Status 404 returned error can't find the container with id 8d846f7c149a36f54b1a266e68623cf1e4fe47ff589deb6bba7efd877cbf00e8
Oct 30 18:46:45.002167 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:45.002147    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-nv4z9"]
Oct 30 18:46:45.009437 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:46:45.009417    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd634ca90_4998_45ba_a69d_276ff6ceae58.slice/crio-a755ad3c7feb62a38eec40d08d6720370c964ebb47799995731bd018b7c5445b WatchSource:0}: Error finding container a755ad3c7feb62a38eec40d08d6720370c964ebb47799995731bd018b7c5445b: Status 404 returned error can't find the container with id a755ad3c7feb62a38eec40d08d6720370c964ebb47799995731bd018b7c5445b
Oct 30 18:46:45.373171 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:45.373111    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-lbktr" event={"ID":"58b38b02-20af-4975-bf9a-54c6a31e8b64","Type":"ContainerStarted","Data":"8d846f7c149a36f54b1a266e68623cf1e4fe47ff589deb6bba7efd877cbf00e8"}
Oct 30 18:46:45.374272 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:45.374251    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-nv4z9" event={"ID":"d634ca90-4998-45ba-a69d-276ff6ceae58","Type":"ContainerStarted","Data":"a755ad3c7feb62a38eec40d08d6720370c964ebb47799995731bd018b7c5445b"}
Oct 30 18:46:47.383349 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:47.383314    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-lbktr" event={"ID":"58b38b02-20af-4975-bf9a-54c6a31e8b64","Type":"ContainerStarted","Data":"77d53887f03a2224f4909fffcf54f1f2f8e93ecc7b0a9e81fa3cbdeb0ca6a72c"}
Oct 30 18:46:47.383724 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:47.383424    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-56ddc4c448-lbktr"
Oct 30 18:46:48.388908 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:48.388876    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-nv4z9" event={"ID":"d634ca90-4998-45ba-a69d-276ff6ceae58","Type":"ContainerStarted","Data":"99466f605500238e08dbea4cc967086e977a01e7099d14f8ded1096a3f25f63e"}
Oct 30 18:46:48.389390 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:48.388959    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7f9f8cdc65-nv4z9"
Oct 30 18:46:48.501268 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:48.501180    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/model-serving-api-56ddc4c448-lbktr" podStartSLOduration=2.2415909 podStartE2EDuration="4.501167919s" totalImagesPullingTime="2.259577019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:47.496983417 +0000 UTC m=+502.014668316" watchObservedRunningTime="2026-10-30 18:46:48.501167919 +0000 UTC m=+503.018852819"
Oct 30 18:46:48.501268 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:48.501257    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/odh-model-controller-7f9f8cdc65-nv4z9" podStartSLOduration=1.367452387 podStartE2EDuration="4.501253593s" totalImagesPullingTime="3.133801206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:48.500334673 +0000 UTC m=+503.018019581" watchObservedRunningTime="2026-10-30 18:46:48.501253593 +0000 UTC m=+503.018938490"
Oct 30 18:46:58.393363 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:58.393331    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-56ddc4c448-lbktr"
Oct 30 18:46:59.394801 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:46:59.394773    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7f9f8cdc65-nv4z9"
Oct 30 18:47:10.155977 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:47:10.155938    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:47:40.158225 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:47:40.158183    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:48:10.156484 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:48:10.156362    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:48:25.681586 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:25.681554    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 18:48:25.682574 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:25.682554    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 18:48:26.684244 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:26.684207    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-swmzm"]
Oct 30 18:48:26.686373 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:26.686352    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-swmzm"
Oct 30 18:48:26.736095 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:26.736070    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Oct 30 18:48:26.741433 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:26.741411    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-swmzm"]
Oct 30 18:48:26.855251 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:26.855224    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/04283832-2543-44dc-834f-4a8e323d26ee-data\") pod \"seaweedfs-tls-custom-64cc6669cb-swmzm\" (UID: \"04283832-2543-44dc-834f-4a8e323d26ee\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-swmzm"
Oct 30 18:48:26.855377 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:26.855259    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkmq4\" (UniqueName: \"kubernetes.io/projected/04283832-2543-44dc-834f-4a8e323d26ee-kube-api-access-xkmq4\") pod \"seaweedfs-tls-custom-64cc6669cb-swmzm\" (UID: \"04283832-2543-44dc-834f-4a8e323d26ee\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-swmzm"
Oct 30 18:48:26.956083 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:26.956030    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/04283832-2543-44dc-834f-4a8e323d26ee-data\") pod \"seaweedfs-tls-custom-64cc6669cb-swmzm\" (UID: \"04283832-2543-44dc-834f-4a8e323d26ee\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-swmzm"
Oct 30 18:48:26.956083 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:26.956068    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkmq4\" (UniqueName: \"kubernetes.io/projected/04283832-2543-44dc-834f-4a8e323d26ee-kube-api-access-xkmq4\") pod \"seaweedfs-tls-custom-64cc6669cb-swmzm\" (UID: \"04283832-2543-44dc-834f-4a8e323d26ee\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-swmzm"
Oct 30 18:48:26.956463 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:26.956444    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/04283832-2543-44dc-834f-4a8e323d26ee-data\") pod \"seaweedfs-tls-custom-64cc6669cb-swmzm\" (UID: \"04283832-2543-44dc-834f-4a8e323d26ee\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-swmzm"
Oct 30 18:48:27.014798 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:27.014778    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkmq4\" (UniqueName: \"kubernetes.io/projected/04283832-2543-44dc-834f-4a8e323d26ee-kube-api-access-xkmq4\") pod \"seaweedfs-tls-custom-64cc6669cb-swmzm\" (UID: \"04283832-2543-44dc-834f-4a8e323d26ee\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-swmzm"
Oct 30 18:48:27.297038 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:27.297015    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-swmzm"
Oct 30 18:48:27.665173 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:27.665091    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-swmzm"]
Oct 30 18:48:27.672956 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:48:27.672933    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04283832_2543_44dc_834f_4a8e323d26ee.slice/crio-7bba7cedb18b5f2a2e649abeb182282572e41842c5671d4b893b8749eae33e57 WatchSource:0}: Error finding container 7bba7cedb18b5f2a2e649abeb182282572e41842c5671d4b893b8749eae33e57: Status 404 returned error can't find the container with id 7bba7cedb18b5f2a2e649abeb182282572e41842c5671d4b893b8749eae33e57
Oct 30 18:48:27.725758 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:27.725729    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-swmzm" event={"ID":"04283832-2543-44dc-834f-4a8e323d26ee","Type":"ContainerStarted","Data":"7bba7cedb18b5f2a2e649abeb182282572e41842c5671d4b893b8749eae33e57"}
Oct 30 18:48:28.731223 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:28.731119    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-swmzm" event={"ID":"04283832-2543-44dc-834f-4a8e323d26ee","Type":"ContainerStarted","Data":"295ed5649eced32cd1a1f1a84b83e80a03cbb8371e68c80d3f64832d4916ed31"}
Oct 30 18:48:28.843960 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:28.843904    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-64cc6669cb-swmzm" podStartSLOduration=2.085032777 podStartE2EDuration="2.843892414s" totalImagesPullingTime="758.859637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:28.843281201 +0000 UTC m=+603.360966100" watchObservedRunningTime="2026-10-30 18:48:28.843892414 +0000 UTC m=+603.361577313"
Oct 30 18:48:30.104094 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:30.104058    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-swmzm"]
Oct 30 18:48:30.738267 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:30.738227    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-64cc6669cb-swmzm" podUID="04283832-2543-44dc-834f-4a8e323d26ee" containerName="seaweedfs-tls-custom" containerID="cri-o://295ed5649eced32cd1a1f1a84b83e80a03cbb8371e68c80d3f64832d4916ed31" gracePeriod=30
Oct 30 18:48:40.157253 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:48:40.157217    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:48:59.073517 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.073490    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-swmzm"
Oct 30 18:48:59.169982 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.169956    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04283832-2543-44dc-834f-4a8e323d26ee-data\" (UniqueName: \"kubernetes.io/empty-dir/04283832-2543-44dc-834f-4a8e323d26ee-data\") pod \"04283832-2543-44dc-834f-4a8e323d26ee\" (UID: \"04283832-2543-44dc-834f-4a8e323d26ee\") "
Oct 30 18:48:59.170119 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.169995    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04283832-2543-44dc-834f-4a8e323d26ee-kube-api-access-xkmq4\" (UniqueName: \"kubernetes.io/projected/04283832-2543-44dc-834f-4a8e323d26ee-kube-api-access-xkmq4\") pod \"04283832-2543-44dc-834f-4a8e323d26ee\" (UID: \"04283832-2543-44dc-834f-4a8e323d26ee\") "
Oct 30 18:48:59.171270 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.171245    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04283832-2543-44dc-834f-4a8e323d26ee-data" pod "04283832-2543-44dc-834f-4a8e323d26ee" (UID: "04283832-2543-44dc-834f-4a8e323d26ee"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:48:59.172086 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.172062    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04283832-2543-44dc-834f-4a8e323d26ee-kube-api-access-xkmq4" pod "04283832-2543-44dc-834f-4a8e323d26ee" (UID: "04283832-2543-44dc-834f-4a8e323d26ee"). InnerVolumeSpecName "kube-api-access-xkmq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:59.231258 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.231189    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-685c9f986c-b5l52"]
Oct 30 18:48:59.231541 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.231524    4511 cpu_manager.go:414] "containerMap: removing container" podUID="04283832-2543-44dc-834f-4a8e323d26ee" containerName="seaweedfs-tls-custom"
Oct 30 18:48:59.231618 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.231543    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04283832-2543-44dc-834f-4a8e323d26ee" containerName="seaweedfs-tls-custom"
Oct 30 18:48:59.231679 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.231634    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="04283832-2543-44dc-834f-4a8e323d26ee" containerName="seaweedfs-tls-custom"
Oct 30 18:48:59.235923 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.235902    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-685c9f986c-b5l52"
Oct 30 18:48:59.271058 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.271035    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/408366ab-3633-40e4-9a40-8d58b9e95490-data\") pod \"seaweedfs-tls-custom-685c9f986c-b5l52\" (UID: \"408366ab-3633-40e4-9a40-8d58b9e95490\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-b5l52"
Oct 30 18:48:59.271168 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.271065    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r4mp\" (UniqueName: \"kubernetes.io/projected/408366ab-3633-40e4-9a40-8d58b9e95490-kube-api-access-8r4mp\") pod \"seaweedfs-tls-custom-685c9f986c-b5l52\" (UID: \"408366ab-3633-40e4-9a40-8d58b9e95490\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-b5l52"
Oct 30 18:48:59.271228 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.271186    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/408366ab-3633-40e4-9a40-8d58b9e95490-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-685c9f986c-b5l52\" (UID: \"408366ab-3633-40e4-9a40-8d58b9e95490\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-b5l52"
Oct 30 18:48:59.271228 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.271220    4511 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/04283832-2543-44dc-834f-4a8e323d26ee-data\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:59.271296 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.271230    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkmq4\" (UniqueName: \"kubernetes.io/projected/04283832-2543-44dc-834f-4a8e323d26ee-kube-api-access-xkmq4\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:59.289615 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.289595    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-685c9f986c-b5l52"]
Oct 30 18:48:59.290718 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.290701    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Oct 30 18:48:59.372257 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.372232    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/408366ab-3633-40e4-9a40-8d58b9e95490-data\") pod \"seaweedfs-tls-custom-685c9f986c-b5l52\" (UID: \"408366ab-3633-40e4-9a40-8d58b9e95490\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-b5l52"
Oct 30 18:48:59.372369 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.372262    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r4mp\" (UniqueName: \"kubernetes.io/projected/408366ab-3633-40e4-9a40-8d58b9e95490-kube-api-access-8r4mp\") pod \"seaweedfs-tls-custom-685c9f986c-b5l52\" (UID: \"408366ab-3633-40e4-9a40-8d58b9e95490\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-b5l52"
Oct 30 18:48:59.372369 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.372303    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/408366ab-3633-40e4-9a40-8d58b9e95490-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-685c9f986c-b5l52\" (UID: \"408366ab-3633-40e4-9a40-8d58b9e95490\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-b5l52"
Oct 30 18:48:59.372601 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.372584    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/408366ab-3633-40e4-9a40-8d58b9e95490-data\") pod \"seaweedfs-tls-custom-685c9f986c-b5l52\" (UID: \"408366ab-3633-40e4-9a40-8d58b9e95490\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-b5l52"
Oct 30 18:48:59.374667 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.374651    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/408366ab-3633-40e4-9a40-8d58b9e95490-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-685c9f986c-b5l52\" (UID: \"408366ab-3633-40e4-9a40-8d58b9e95490\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-b5l52"
Oct 30 18:48:59.430474 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.430449    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r4mp\" (UniqueName: \"kubernetes.io/projected/408366ab-3633-40e4-9a40-8d58b9e95490-kube-api-access-8r4mp\") pod \"seaweedfs-tls-custom-685c9f986c-b5l52\" (UID: \"408366ab-3633-40e4-9a40-8d58b9e95490\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-b5l52"
Oct 30 18:48:59.546465 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.546434    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-685c9f986c-b5l52"
Oct 30 18:48:59.833602 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.833543    4511 generic.go:356] "Generic (PLEG): container finished" podID="04283832-2543-44dc-834f-4a8e323d26ee" containerID="295ed5649eced32cd1a1f1a84b83e80a03cbb8371e68c80d3f64832d4916ed31" exitCode=0
Oct 30 18:48:59.833602 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.833588    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-swmzm"
Oct 30 18:48:59.833767 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.833630    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-swmzm" event={"ID":"04283832-2543-44dc-834f-4a8e323d26ee","Type":"ContainerDied","Data":"295ed5649eced32cd1a1f1a84b83e80a03cbb8371e68c80d3f64832d4916ed31"}
Oct 30 18:48:59.833767 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.833679    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-swmzm" event={"ID":"04283832-2543-44dc-834f-4a8e323d26ee","Type":"ContainerDied","Data":"7bba7cedb18b5f2a2e649abeb182282572e41842c5671d4b893b8749eae33e57"}
Oct 30 18:48:59.833767 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.833704    4511 scope.go:122] "RemoveContainer" containerID="295ed5649eced32cd1a1f1a84b83e80a03cbb8371e68c80d3f64832d4916ed31"
Oct 30 18:48:59.842667 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.842651    4511 scope.go:122] "RemoveContainer" containerID="295ed5649eced32cd1a1f1a84b83e80a03cbb8371e68c80d3f64832d4916ed31"
Oct 30 18:48:59.842914 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:48:59.842892    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"295ed5649eced32cd1a1f1a84b83e80a03cbb8371e68c80d3f64832d4916ed31\": container with ID starting with 295ed5649eced32cd1a1f1a84b83e80a03cbb8371e68c80d3f64832d4916ed31 not found: ID does not exist" containerID="295ed5649eced32cd1a1f1a84b83e80a03cbb8371e68c80d3f64832d4916ed31"
Oct 30 18:48:59.842968 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.842924    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"295ed5649eced32cd1a1f1a84b83e80a03cbb8371e68c80d3f64832d4916ed31"} err="failed to get container status \"295ed5649eced32cd1a1f1a84b83e80a03cbb8371e68c80d3f64832d4916ed31\": rpc error: code = NotFound desc = could not find container \"295ed5649eced32cd1a1f1a84b83e80a03cbb8371e68c80d3f64832d4916ed31\": container with ID starting with 295ed5649eced32cd1a1f1a84b83e80a03cbb8371e68c80d3f64832d4916ed31 not found: ID does not exist"
Oct 30 18:48:59.913077 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.913057    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-685c9f986c-b5l52"]
Oct 30 18:48:59.921799 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:48:59.921776    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408366ab_3633_40e4_9a40_8d58b9e95490.slice/crio-3d31400bbd3b1d08d8a4b63f2952acd7d6ab7a372c802c578b98c9272cbfeaaa WatchSource:0}: Error finding container 3d31400bbd3b1d08d8a4b63f2952acd7d6ab7a372c802c578b98c9272cbfeaaa: Status 404 returned error can't find the container with id 3d31400bbd3b1d08d8a4b63f2952acd7d6ab7a372c802c578b98c9272cbfeaaa
Oct 30 18:48:59.923349 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.923333    4511 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:48:59.988911 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.988889    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-swmzm"]
Oct 30 18:48:59.993068 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:48:59.993050    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-swmzm"]
Oct 30 18:49:00.839782 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:00.839751    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-685c9f986c-b5l52" event={"ID":"408366ab-3633-40e4-9a40-8d58b9e95490","Type":"ContainerStarted","Data":"2eb111bbdcaf8a9707a4530fc7ee0189248f10e9af145c29df2ff31ddb29aba2"}
Oct 30 18:49:00.840084 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:00.839850    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-685c9f986c-b5l52" event={"ID":"408366ab-3633-40e4-9a40-8d58b9e95490","Type":"ContainerStarted","Data":"3d31400bbd3b1d08d8a4b63f2952acd7d6ab7a372c802c578b98c9272cbfeaaa"}
Oct 30 18:49:00.950221 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:00.950173    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-685c9f986c-b5l52" podStartSLOduration=1.095743832 podStartE2EDuration="1.9501603s" totalImagesPullingTime="854.416468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:00.949402806 +0000 UTC m=+635.467087708" watchObservedRunningTime="2026-10-30 18:49:00.9501603 +0000 UTC m=+635.467845224"
Oct 30 18:49:01.655799 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:01.655768    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04283832-2543-44dc-834f-4a8e323d26ee" path="/var/lib/kubelet/pods/04283832-2543-44dc-834f-4a8e323d26ee/volumes"
Oct 30 18:49:10.162184 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:49:10.162145    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:49:14.667239 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:14.667203    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-v9m85"]
Oct 30 18:49:14.670738 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:14.670718    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-5c74974785-v9m85"
Oct 30 18:49:14.724854 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:14.724821    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Oct 30 18:49:14.727425 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:14.727400    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Oct 30 18:49:14.773464 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:14.773432    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-v9m85"]
Oct 30 18:49:14.784553 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:14.784515    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa-data\") pod \"seaweedfs-tls-serving-5c74974785-v9m85\" (UID: \"6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa\") " pod="kserve/seaweedfs-tls-serving-5c74974785-v9m85"
Oct 30 18:49:14.784553 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:14.784549    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq5xg\" (UniqueName: \"kubernetes.io/projected/6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa-kube-api-access-hq5xg\") pod \"seaweedfs-tls-serving-5c74974785-v9m85\" (UID: \"6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa\") " pod="kserve/seaweedfs-tls-serving-5c74974785-v9m85"
Oct 30 18:49:14.784723 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:14.784630    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-v9m85\" (UID: \"6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa\") " pod="kserve/seaweedfs-tls-serving-5c74974785-v9m85"
Oct 30 18:49:14.885661 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:14.885628    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa-data\") pod \"seaweedfs-tls-serving-5c74974785-v9m85\" (UID: \"6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa\") " pod="kserve/seaweedfs-tls-serving-5c74974785-v9m85"
Oct 30 18:49:14.885661 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:14.885664    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5xg\" (UniqueName: \"kubernetes.io/projected/6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa-kube-api-access-hq5xg\") pod \"seaweedfs-tls-serving-5c74974785-v9m85\" (UID: \"6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa\") " pod="kserve/seaweedfs-tls-serving-5c74974785-v9m85"
Oct 30 18:49:14.885869 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:14.885690    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-v9m85\" (UID: \"6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa\") " pod="kserve/seaweedfs-tls-serving-5c74974785-v9m85"
Oct 30 18:49:14.886159 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:14.886108    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa-data\") pod \"seaweedfs-tls-serving-5c74974785-v9m85\" (UID: \"6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa\") " pod="kserve/seaweedfs-tls-serving-5c74974785-v9m85"
Oct 30 18:49:14.888289 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:14.888267    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-v9m85\" (UID: \"6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa\") " pod="kserve/seaweedfs-tls-serving-5c74974785-v9m85"
Oct 30 18:49:14.945026 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:14.944968    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq5xg\" (UniqueName: \"kubernetes.io/projected/6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa-kube-api-access-hq5xg\") pod \"seaweedfs-tls-serving-5c74974785-v9m85\" (UID: \"6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa\") " pod="kserve/seaweedfs-tls-serving-5c74974785-v9m85"
Oct 30 18:49:14.981913 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:14.981890    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-5c74974785-v9m85"
Oct 30 18:49:15.347210 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:15.347182    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-v9m85"]
Oct 30 18:49:15.355646 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:49:15.355616    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8b2cc6_8eb9_41f5_8396_d0e845c10dfa.slice/crio-d11ffc6da0296bf6a9fb909352d31686059d9ef8b40d9a1179fd4647eb0dfc8d WatchSource:0}: Error finding container d11ffc6da0296bf6a9fb909352d31686059d9ef8b40d9a1179fd4647eb0dfc8d: Status 404 returned error can't find the container with id d11ffc6da0296bf6a9fb909352d31686059d9ef8b40d9a1179fd4647eb0dfc8d
Oct 30 18:49:15.896161 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:15.896114    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-5c74974785-v9m85" event={"ID":"6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa","Type":"ContainerStarted","Data":"d11ffc6da0296bf6a9fb909352d31686059d9ef8b40d9a1179fd4647eb0dfc8d"}
Oct 30 18:49:16.900803 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:16.900769    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-5c74974785-v9m85" event={"ID":"6b8b2cc6-8eb9-41f5-8396-d0e845c10dfa","Type":"ContainerStarted","Data":"7163c601aa87b34b6fb7bf0a5db0f969289f11535192a453b33ea43ef06c8b4c"}
Oct 30 18:49:17.011574 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:17.011530    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-5c74974785-v9m85" podStartSLOduration=2.239429253 podStartE2EDuration="3.011517729s" totalImagesPullingTime="772.088476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:17.011345776 +0000 UTC m=+651.529030675" watchObservedRunningTime="2026-10-30 18:49:17.011517729 +0000 UTC m=+651.529202628"
Oct 30 18:49:40.156012 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:49:40.155927    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:49:50.989687 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:50.989655    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"]
Oct 30 18:49:50.995068 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:50.995049    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:49:51.045028 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.044998    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 18:49:51.092022 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.091995    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 18:49:51.092022 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.092018    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Oct 30 18:49:51.092272 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.092087    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 18:49:51.092532 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.092506    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 18:49:51.092706 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.092687    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 18:49:51.092819 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.092689    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Oct 30 18:49:51.095890 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.095866    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"]
Oct 30 18:49:51.148797 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.148767    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") " pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:49:51.148921 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.148804    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") " pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:49:51.148921 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.148901    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") " pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:49:51.148990 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.148973    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") " pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:49:51.149038 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.148996    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7dgs\" (UniqueName: \"kubernetes.io/projected/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kube-api-access-m7dgs\") pod \"isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") " pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:49:51.249825 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.249748    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") " pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:49:51.249825 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.249805    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") " pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:49:51.250037 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.249925    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7dgs\" (UniqueName: \"kubernetes.io/projected/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kube-api-access-m7dgs\") pod \"isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") " pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:49:51.250037 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.249969    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") " pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:49:51.250037 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.250003    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") " pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:49:51.250225 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.250153    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") " pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:49:51.250734 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.250672    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") " pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:49:51.250734 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.250676    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") " pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:49:51.252532 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.252510    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") " pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:49:51.318756 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.318727    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7dgs\" (UniqueName: \"kubernetes.io/projected/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kube-api-access-m7dgs\") pod \"isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") " pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:49:51.612430 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:51.612398    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:49:52.037329 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:52.037250    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"]
Oct 30 18:49:52.044862 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:49:52.044837    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041f5b38_a72c_4d6b_b838_fc7440a6d4fd.slice/crio-7f8376be0cf277fae5fd0666c7e2ceaa052ebb6f1a4093d5c47be6cf230e655e WatchSource:0}: Error finding container 7f8376be0cf277fae5fd0666c7e2ceaa052ebb6f1a4093d5c47be6cf230e655e: Status 404 returned error can't find the container with id 7f8376be0cf277fae5fd0666c7e2ceaa052ebb6f1a4093d5c47be6cf230e655e
Oct 30 18:49:53.031060 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:53.031023    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" event={"ID":"041f5b38-a72c-4d6b-b838-fc7440a6d4fd","Type":"ContainerStarted","Data":"7f8376be0cf277fae5fd0666c7e2ceaa052ebb6f1a4093d5c47be6cf230e655e"}
Oct 30 18:49:57.048960 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:49:57.048927    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" event={"ID":"041f5b38-a72c-4d6b-b838-fc7440a6d4fd","Type":"ContainerStarted","Data":"f729d5f7343f2a1a342bdc3f53d569b53682502cefc7dd99ff70373dce6ba64a"}
Oct 30 18:50:00.060713 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:50:00.060683    4511 generic.go:356] "Generic (PLEG): container finished" podID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerID="f729d5f7343f2a1a342bdc3f53d569b53682502cefc7dd99ff70373dce6ba64a" exitCode=0
Oct 30 18:50:00.061025 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:50:00.060750    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" event={"ID":"041f5b38-a72c-4d6b-b838-fc7440a6d4fd","Type":"ContainerDied","Data":"f729d5f7343f2a1a342bdc3f53d569b53682502cefc7dd99ff70373dce6ba64a"}
Oct 30 18:50:10.103656 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:50:10.103625    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" event={"ID":"041f5b38-a72c-4d6b-b838-fc7440a6d4fd","Type":"ContainerStarted","Data":"0477a4f0a1fe080f5d3162d4f765b18fae28e6506dc55cbaf98e42d7f083c516"}
Oct 30 18:50:10.156401 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:50:10.156367    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:50:15.123863 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:50:15.123819    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" event={"ID":"041f5b38-a72c-4d6b-b838-fc7440a6d4fd","Type":"ContainerStarted","Data":"22b8be52e4e3aca5d98a17bf292866f661dc2c99cef1ad4147735d71cf67580d"}
Oct 30 18:50:15.124449 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:50:15.124037    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:50:15.124449 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:50:15.124111    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:50:15.125680 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:50:15.125647    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Oct 30 18:50:15.286264 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:50:15.286170    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" podStartSLOduration=2.695633718 podStartE2EDuration="25.286156919s" totalImagesPullingTime="18.856564122s" totalInitContainerRuntime="3.733959079s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:15.285465059 +0000 UTC m=+709.803149962" watchObservedRunningTime="2026-10-30 18:50:15.286156919 +0000 UTC m=+709.803841813"
Oct 30 18:50:16.127650 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:50:16.127611    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Oct 30 18:50:21.133949 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:50:21.133919    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:50:21.134519 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:50:21.134436    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Oct 30 18:50:31.134638 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:50:31.134595    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Oct 30 18:50:40.156502 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:50:40.156466    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:50:41.134732 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:50:41.134695    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Oct 30 18:50:51.134772 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:50:51.134731    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Oct 30 18:51:01.134960 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:01.134924    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Oct 30 18:51:10.156039 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:51:10.155960    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:51:11.135054 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:11.135012    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Oct 30 18:51:21.135188 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:21.135148    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Oct 30 18:51:31.134533 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:31.134478    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Oct 30 18:51:39.655981 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:39.655955    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:51:40.156069 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:51:40.156036    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:51:51.262747 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:51.262707    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"]
Oct 30 18:51:51.263300 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:51.263015    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kserve-container" containerID="cri-o://0477a4f0a1fe080f5d3162d4f765b18fae28e6506dc55cbaf98e42d7f083c516" gracePeriod=30
Oct 30 18:51:51.263300 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:51.263041    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kube-rbac-proxy" containerID="cri-o://22b8be52e4e3aca5d98a17bf292866f661dc2c99cef1ad4147735d71cf67580d" gracePeriod=30
Oct 30 18:51:51.464843 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:51.464817    4511 generic.go:356] "Generic (PLEG): container finished" podID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerID="22b8be52e4e3aca5d98a17bf292866f661dc2c99cef1ad4147735d71cf67580d" exitCode=0
Oct 30 18:51:51.464954 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:51.464889    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" event={"ID":"041f5b38-a72c-4d6b-b838-fc7440a6d4fd","Type":"ContainerDied","Data":"22b8be52e4e3aca5d98a17bf292866f661dc2c99cef1ad4147735d71cf67580d"}
Oct 30 18:51:54.801697 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:54.801675    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:51:54.869909 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:54.869884    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kube-api-access-m7dgs\" (UniqueName: \"kubernetes.io/projected/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kube-api-access-m7dgs\") pod \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") "
Oct 30 18:51:54.870016 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:54.869924    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kube-rbac-proxy-sar-config\") pod \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") "
Oct 30 18:51:54.870016 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:54.869953    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-cabundle-cert\") pod \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") "
Oct 30 18:51:54.870083 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:54.870060    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-proxy-tls\" (UniqueName: \"kubernetes.io/secret/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-proxy-tls\") pod \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") "
Oct 30 18:51:54.870120 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:54.870095    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kserve-provision-location\") pod \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\" (UID: \"041f5b38-a72c-4d6b-b838-fc7440a6d4fd\") "
Oct 30 18:51:54.870319 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:54.870289    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kube-rbac-proxy-sar-config" pod "041f5b38-a72c-4d6b-b838-fc7440a6d4fd" (UID: "041f5b38-a72c-4d6b-b838-fc7440a6d4fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:54.870435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:54.870295    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-cabundle-cert" pod "041f5b38-a72c-4d6b-b838-fc7440a6d4fd" (UID: "041f5b38-a72c-4d6b-b838-fc7440a6d4fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:54.870435 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:54.870367    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:54.870568 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:54.870537    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kserve-provision-location" pod "041f5b38-a72c-4d6b-b838-fc7440a6d4fd" (UID: "041f5b38-a72c-4d6b-b838-fc7440a6d4fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:54.872238 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:54.872215    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kube-api-access-m7dgs" pod "041f5b38-a72c-4d6b-b838-fc7440a6d4fd" (UID: "041f5b38-a72c-4d6b-b838-fc7440a6d4fd"). InnerVolumeSpecName "kube-api-access-m7dgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:54.872436 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:54.872416    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-proxy-tls" pod "041f5b38-a72c-4d6b-b838-fc7440a6d4fd" (UID: "041f5b38-a72c-4d6b-b838-fc7440a6d4fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:54.970961 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:54.970937    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7dgs\" (UniqueName: \"kubernetes.io/projected/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kube-api-access-m7dgs\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:54.970961 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:54.970960    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:54.971093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:54.970971    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:54.971093 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:54.970979    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/041f5b38-a72c-4d6b-b838-fc7440a6d4fd-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:55.481198 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:55.481160    4511 generic.go:356] "Generic (PLEG): container finished" podID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerID="0477a4f0a1fe080f5d3162d4f765b18fae28e6506dc55cbaf98e42d7f083c516" exitCode=0
Oct 30 18:51:55.481395 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:55.481222    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" event={"ID":"041f5b38-a72c-4d6b-b838-fc7440a6d4fd","Type":"ContainerDied","Data":"0477a4f0a1fe080f5d3162d4f765b18fae28e6506dc55cbaf98e42d7f083c516"}
Oct 30 18:51:55.481395 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:55.481288    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v" event={"ID":"041f5b38-a72c-4d6b-b838-fc7440a6d4fd","Type":"ContainerDied","Data":"7f8376be0cf277fae5fd0666c7e2ceaa052ebb6f1a4093d5c47be6cf230e655e"}
Oct 30 18:51:55.481395 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:55.481290    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"
Oct 30 18:51:55.481395 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:55.481322    4511 scope.go:122] "RemoveContainer" containerID="22b8be52e4e3aca5d98a17bf292866f661dc2c99cef1ad4147735d71cf67580d"
Oct 30 18:51:55.490996 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:55.490961    4511 scope.go:122] "RemoveContainer" containerID="0477a4f0a1fe080f5d3162d4f765b18fae28e6506dc55cbaf98e42d7f083c516"
Oct 30 18:51:55.499208 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:55.499187    4511 scope.go:122] "RemoveContainer" containerID="f729d5f7343f2a1a342bdc3f53d569b53682502cefc7dd99ff70373dce6ba64a"
Oct 30 18:51:55.506581 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:55.506560    4511 scope.go:122] "RemoveContainer" containerID="22b8be52e4e3aca5d98a17bf292866f661dc2c99cef1ad4147735d71cf67580d"
Oct 30 18:51:55.506860 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:51:55.506840    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b8be52e4e3aca5d98a17bf292866f661dc2c99cef1ad4147735d71cf67580d\": container with ID starting with 22b8be52e4e3aca5d98a17bf292866f661dc2c99cef1ad4147735d71cf67580d not found: ID does not exist" containerID="22b8be52e4e3aca5d98a17bf292866f661dc2c99cef1ad4147735d71cf67580d"
Oct 30 18:51:55.506923 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:55.506872    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b8be52e4e3aca5d98a17bf292866f661dc2c99cef1ad4147735d71cf67580d"} err="failed to get container status \"22b8be52e4e3aca5d98a17bf292866f661dc2c99cef1ad4147735d71cf67580d\": rpc error: code = NotFound desc = could not find container \"22b8be52e4e3aca5d98a17bf292866f661dc2c99cef1ad4147735d71cf67580d\": container with ID starting with 22b8be52e4e3aca5d98a17bf292866f661dc2c99cef1ad4147735d71cf67580d not found: ID does not exist"
Oct 30 18:51:55.506923 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:55.506891    4511 scope.go:122] "RemoveContainer" containerID="0477a4f0a1fe080f5d3162d4f765b18fae28e6506dc55cbaf98e42d7f083c516"
Oct 30 18:51:55.507198 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:51:55.507183    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0477a4f0a1fe080f5d3162d4f765b18fae28e6506dc55cbaf98e42d7f083c516\": container with ID starting with 0477a4f0a1fe080f5d3162d4f765b18fae28e6506dc55cbaf98e42d7f083c516 not found: ID does not exist" containerID="0477a4f0a1fe080f5d3162d4f765b18fae28e6506dc55cbaf98e42d7f083c516"
Oct 30 18:51:55.507240 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:55.507203    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0477a4f0a1fe080f5d3162d4f765b18fae28e6506dc55cbaf98e42d7f083c516"} err="failed to get container status \"0477a4f0a1fe080f5d3162d4f765b18fae28e6506dc55cbaf98e42d7f083c516\": rpc error: code = NotFound desc = could not find container \"0477a4f0a1fe080f5d3162d4f765b18fae28e6506dc55cbaf98e42d7f083c516\": container with ID starting with 0477a4f0a1fe080f5d3162d4f765b18fae28e6506dc55cbaf98e42d7f083c516 not found: ID does not exist"
Oct 30 18:51:55.507240 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:55.507226    4511 scope.go:122] "RemoveContainer" containerID="f729d5f7343f2a1a342bdc3f53d569b53682502cefc7dd99ff70373dce6ba64a"
Oct 30 18:51:55.507471 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:51:55.507452    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f729d5f7343f2a1a342bdc3f53d569b53682502cefc7dd99ff70373dce6ba64a\": container with ID starting with f729d5f7343f2a1a342bdc3f53d569b53682502cefc7dd99ff70373dce6ba64a not found: ID does not exist" containerID="f729d5f7343f2a1a342bdc3f53d569b53682502cefc7dd99ff70373dce6ba64a"
Oct 30 18:51:55.507505 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:55.507475    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f729d5f7343f2a1a342bdc3f53d569b53682502cefc7dd99ff70373dce6ba64a"} err="failed to get container status \"f729d5f7343f2a1a342bdc3f53d569b53682502cefc7dd99ff70373dce6ba64a\": rpc error: code = NotFound desc = could not find container \"f729d5f7343f2a1a342bdc3f53d569b53682502cefc7dd99ff70373dce6ba64a\": container with ID starting with f729d5f7343f2a1a342bdc3f53d569b53682502cefc7dd99ff70373dce6ba64a not found: ID does not exist"
Oct 30 18:51:55.658559 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:55.658531    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"]
Oct 30 18:51:55.704982 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:55.704954    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-pmml-v2-kserve-predictor-7bc9fd5d66-xqq9v"]
Oct 30 18:51:57.656061 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:51:57.656026    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" path="/var/lib/kubelet/pods/041f5b38-a72c-4d6b-b838-fc7440a6d4fd/volumes"
Oct 30 18:52:10.155886 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:52:10.155850    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:52:40.155957 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:52:40.155878    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:53:10.156217 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:53:10.156177    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:53:25.701701 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:53:25.701675    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 18:53:25.704692 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:53:25.704670    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 18:53:40.156402 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:53:40.156354    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:54:10.156827 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:54:10.156737    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:54:34.288956 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.288925    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"]
Oct 30 18:54:34.289429 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.289218    4511 cpu_manager.go:414] "containerMap: removing container" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kserve-container"
Oct 30 18:54:34.289429 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.289228    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kserve-container"
Oct 30 18:54:34.289429 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.289246    4511 cpu_manager.go:414] "containerMap: removing container" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kube-rbac-proxy"
Oct 30 18:54:34.289429 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.289251    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kube-rbac-proxy"
Oct 30 18:54:34.289429 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.289265    4511 cpu_manager.go:414] "containerMap: removing container" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="storage-initializer"
Oct 30 18:54:34.289429 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.289271    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="storage-initializer"
Oct 30 18:54:34.289429 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.289337    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kserve-container"
Oct 30 18:54:34.289429 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.289344    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="041f5b38-a72c-4d6b-b838-fc7440a6d4fd" containerName="kube-rbac-proxy"
Oct 30 18:54:34.292444 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.292429    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:54:34.382477 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.382446    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/997b6acd-c2cf-4208-94a5-87583318be5f-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-84c748568-pgwb2\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:54:34.382477 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.382479    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/997b6acd-c2cf-4208-94a5-87583318be5f-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-84c748568-pgwb2\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:54:34.382645 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.382523    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg5vd\" (UniqueName: \"kubernetes.io/projected/997b6acd-c2cf-4208-94a5-87583318be5f-kube-api-access-sg5vd\") pod \"isvc-predictive-sklearn-predictor-84c748568-pgwb2\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:54:34.382682 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.382656    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/997b6acd-c2cf-4208-94a5-87583318be5f-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-84c748568-pgwb2\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:54:34.382720 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.382694    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/997b6acd-c2cf-4208-94a5-87583318be5f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-84c748568-pgwb2\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:54:34.390397 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.390344    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 18:54:34.390397 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.390379    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 18:54:34.390577 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.390396    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 18:54:34.390577 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.390418    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Oct 30 18:54:34.390577 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.390453    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 18:54:34.390577 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.390552    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 18:54:34.390772 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.390643    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Oct 30 18:54:34.392036 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.392017    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"]
Oct 30 18:54:34.483312 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.483279    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg5vd\" (UniqueName: \"kubernetes.io/projected/997b6acd-c2cf-4208-94a5-87583318be5f-kube-api-access-sg5vd\") pod \"isvc-predictive-sklearn-predictor-84c748568-pgwb2\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:54:34.483468 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.483323    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/997b6acd-c2cf-4208-94a5-87583318be5f-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-84c748568-pgwb2\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:54:34.483468 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.483342    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/997b6acd-c2cf-4208-94a5-87583318be5f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-84c748568-pgwb2\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:54:34.483468 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.483375    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/997b6acd-c2cf-4208-94a5-87583318be5f-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-84c748568-pgwb2\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:54:34.483468 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.483394    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/997b6acd-c2cf-4208-94a5-87583318be5f-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-84c748568-pgwb2\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:54:34.483897 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.483872    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/997b6acd-c2cf-4208-94a5-87583318be5f-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-84c748568-pgwb2\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:54:34.484009 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.483991    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/997b6acd-c2cf-4208-94a5-87583318be5f-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-84c748568-pgwb2\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:54:34.484248 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.484093    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/997b6acd-c2cf-4208-94a5-87583318be5f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-84c748568-pgwb2\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:54:34.485923 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.485897    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/997b6acd-c2cf-4208-94a5-87583318be5f-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-84c748568-pgwb2\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:54:34.542914 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.542849    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg5vd\" (UniqueName: \"kubernetes.io/projected/997b6acd-c2cf-4208-94a5-87583318be5f-kube-api-access-sg5vd\") pod \"isvc-predictive-sklearn-predictor-84c748568-pgwb2\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:54:34.609010 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:34.608985    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:54:35.030606 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:35.030575    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"]
Oct 30 18:54:35.037864 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:54:35.037835    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997b6acd_c2cf_4208_94a5_87583318be5f.slice/crio-3a26fba002d5ce654a35bbfe106f71ef27c2d64c75ccca04cb0d794934f03130 WatchSource:0}: Error finding container 3a26fba002d5ce654a35bbfe106f71ef27c2d64c75ccca04cb0d794934f03130: Status 404 returned error can't find the container with id 3a26fba002d5ce654a35bbfe106f71ef27c2d64c75ccca04cb0d794934f03130
Oct 30 18:54:36.018075 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:36.018039    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" event={"ID":"997b6acd-c2cf-4208-94a5-87583318be5f","Type":"ContainerStarted","Data":"c4dcbf90cb29af9e6fadd49583682165b50fb01fb6a5f3ebbbd1a426f4640b36"}
Oct 30 18:54:36.018532 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:36.018093    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" event={"ID":"997b6acd-c2cf-4208-94a5-87583318be5f","Type":"ContainerStarted","Data":"3a26fba002d5ce654a35bbfe106f71ef27c2d64c75ccca04cb0d794934f03130"}
Oct 30 18:54:39.030313 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:39.030282    4511 generic.go:356] "Generic (PLEG): container finished" podID="997b6acd-c2cf-4208-94a5-87583318be5f" containerID="c4dcbf90cb29af9e6fadd49583682165b50fb01fb6a5f3ebbbd1a426f4640b36" exitCode=0
Oct 30 18:54:39.030685 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:39.030319    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" event={"ID":"997b6acd-c2cf-4208-94a5-87583318be5f","Type":"ContainerDied","Data":"c4dcbf90cb29af9e6fadd49583682165b50fb01fb6a5f3ebbbd1a426f4640b36"}
Oct 30 18:54:39.031552 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:54:39.031536    4511 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:54:40.156462 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:54:40.156416    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:55:06.153952 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:06.153925    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" event={"ID":"997b6acd-c2cf-4208-94a5-87583318be5f","Type":"ContainerStarted","Data":"c682d1f5f0581ba3fb78d86e570d94d8a0c1689a785e87ef14f465ede018d599"}
Oct 30 18:55:06.154391 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:06.153982    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" event={"ID":"997b6acd-c2cf-4208-94a5-87583318be5f","Type":"ContainerStarted","Data":"a1cfe3a9bfc5f1e9cc79b7ccfb202f311b8ef6df12548ea9d8bc10d99aaa8b81"}
Oct 30 18:55:07.158112 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:07.158077    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:55:07.320832 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:07.320774    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" podStartSLOduration=2.665992046 podStartE2EDuration="33.320760526s" totalImagesPullingTime="26.906129085s" totalInitContainerRuntime="3.748639395s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:07.319401526 +0000 UTC m=+1001.837086426" watchObservedRunningTime="2026-10-30 18:55:07.320760526 +0000 UTC m=+1001.838445466"
Oct 30 18:55:08.161931 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:08.161894    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:55:08.163186 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:08.163165    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Oct 30 18:55:09.166228 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:09.166184    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Oct 30 18:55:10.156183 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:55:10.156144    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:55:14.170544 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:14.170511    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:55:14.171163 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:14.171120    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Oct 30 18:55:24.171913 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:24.171876    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Oct 30 18:55:26.929931 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:26.929894    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-54kmx-zspq4"]
Oct 30 18:55:26.932542 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:26.932526    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-54kmx-zspq4"
Oct 30 18:55:26.982356 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:26.982335    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 18:55:27.007808 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:27.007783    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-service-ca-bundle\") pod \"periodic-gathering-54kmx-zspq4\" (UID: \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\") " pod="openshift-insights/periodic-gathering-54kmx-zspq4"
Oct 30 18:55:27.007930 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:27.007812    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-trusted-ca-bundle\") pod \"periodic-gathering-54kmx-zspq4\" (UID: \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\") " pod="openshift-insights/periodic-gathering-54kmx-zspq4"
Oct 30 18:55:27.007930 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:27.007849    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-archives-path\") pod \"periodic-gathering-54kmx-zspq4\" (UID: \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\") " pod="openshift-insights/periodic-gathering-54kmx-zspq4"
Oct 30 18:55:27.007930 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:27.007911    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvqfz\" (UniqueName: \"kubernetes.io/projected/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-kube-api-access-hvqfz\") pod \"periodic-gathering-54kmx-zspq4\" (UID: \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\") " pod="openshift-insights/periodic-gathering-54kmx-zspq4"
Oct 30 18:55:27.077242 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:27.077215    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-z8gkb\""
Oct 30 18:55:27.108454 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:27.108427    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-service-ca-bundle\") pod \"periodic-gathering-54kmx-zspq4\" (UID: \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\") " pod="openshift-insights/periodic-gathering-54kmx-zspq4"
Oct 30 18:55:27.108577 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:27.108460    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-trusted-ca-bundle\") pod \"periodic-gathering-54kmx-zspq4\" (UID: \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\") " pod="openshift-insights/periodic-gathering-54kmx-zspq4"
Oct 30 18:55:27.108577 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:27.108494    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-archives-path\") pod \"periodic-gathering-54kmx-zspq4\" (UID: \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\") " pod="openshift-insights/periodic-gathering-54kmx-zspq4"
Oct 30 18:55:27.108577 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:27.108512    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvqfz\" (UniqueName: \"kubernetes.io/projected/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-kube-api-access-hvqfz\") pod \"periodic-gathering-54kmx-zspq4\" (UID: \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\") " pod="openshift-insights/periodic-gathering-54kmx-zspq4"
Oct 30 18:55:27.108923 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:27.108901    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-archives-path\") pod \"periodic-gathering-54kmx-zspq4\" (UID: \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\") " pod="openshift-insights/periodic-gathering-54kmx-zspq4"
Oct 30 18:55:27.109151 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:27.109107    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-service-ca-bundle\") pod \"periodic-gathering-54kmx-zspq4\" (UID: \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\") " pod="openshift-insights/periodic-gathering-54kmx-zspq4"
Oct 30 18:55:27.125870 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:27.125850    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 18:55:27.126924 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:27.126906    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-54kmx-zspq4"]
Oct 30 18:55:27.129454 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:27.129430    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-trusted-ca-bundle\") pod \"periodic-gathering-54kmx-zspq4\" (UID: \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\") " pod="openshift-insights/periodic-gathering-54kmx-zspq4"
Oct 30 18:55:27.174918 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:27.174889    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvqfz\" (UniqueName: \"kubernetes.io/projected/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-kube-api-access-hvqfz\") pod \"periodic-gathering-54kmx-zspq4\" (UID: \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\") " pod="openshift-insights/periodic-gathering-54kmx-zspq4"
Oct 30 18:55:27.243078 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:27.243023    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-54kmx-zspq4"
Oct 30 18:55:27.608831 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:27.608799    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-54kmx-zspq4"]
Oct 30 18:55:27.616439 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:55:27.616410    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc065bfdd_83bb_4ea8_b42a_6fd9c18f4d3c.slice/crio-7d922f7078ccb2c129fa8ea60d019cde8d9fd2eca48dc1ce86e6e1358340b910 WatchSource:0}: Error finding container 7d922f7078ccb2c129fa8ea60d019cde8d9fd2eca48dc1ce86e6e1358340b910: Status 404 returned error can't find the container with id 7d922f7078ccb2c129fa8ea60d019cde8d9fd2eca48dc1ce86e6e1358340b910
Oct 30 18:55:28.236262 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:28.236223    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-54kmx-zspq4" event={"ID":"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c","Type":"ContainerStarted","Data":"f65a05038656676e5ace29ab4b4259e4ab9b34ae0bdeb1cfe2e3779f9d774423"}
Oct 30 18:55:28.236672 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:28.236275    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-54kmx-zspq4" event={"ID":"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c","Type":"ContainerStarted","Data":"7d922f7078ccb2c129fa8ea60d019cde8d9fd2eca48dc1ce86e6e1358340b910"}
Oct 30 18:55:28.346912 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:28.346853    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-54kmx-zspq4" podStartSLOduration=2.346840365 podStartE2EDuration="2.346840365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:28.34672016 +0000 UTC m=+1022.864405063" watchObservedRunningTime="2026-10-30 18:55:28.346840365 +0000 UTC m=+1022.864525265"
Oct 30 18:55:34.171145 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:34.171085    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Oct 30 18:55:40.156025 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:55:40.155941    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:55:44.171249 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:44.171211    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Oct 30 18:55:54.171203 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:55:54.171165    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Oct 30 18:56:04.171593 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:04.171554    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Oct 30 18:56:10.156635 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:56:10.156594    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:56:14.172315 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:14.172286    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:56:24.794207 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:24.794178    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"]
Oct 30 18:56:24.794620 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:24.794414    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="kserve-container" containerID="cri-o://a1cfe3a9bfc5f1e9cc79b7ccfb202f311b8ef6df12548ea9d8bc10d99aaa8b81" gracePeriod=30
Oct 30 18:56:24.794620 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:24.794444    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="kube-rbac-proxy" containerID="cri-o://c682d1f5f0581ba3fb78d86e570d94d8a0c1689a785e87ef14f465ede018d599" gracePeriod=30
Oct 30 18:56:25.455527 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:25.455499    4511 generic.go:356] "Generic (PLEG): container finished" podID="997b6acd-c2cf-4208-94a5-87583318be5f" containerID="c682d1f5f0581ba3fb78d86e570d94d8a0c1689a785e87ef14f465ede018d599" exitCode=0
Oct 30 18:56:25.455700 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:25.455575    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" event={"ID":"997b6acd-c2cf-4208-94a5-87583318be5f","Type":"ContainerDied","Data":"c682d1f5f0581ba3fb78d86e570d94d8a0c1689a785e87ef14f465ede018d599"}
Oct 30 18:56:28.854043 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:56:28.854009    4511 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997b6acd_c2cf_4208_94a5_87583318be5f.slice/crio-a1cfe3a9bfc5f1e9cc79b7ccfb202f311b8ef6df12548ea9d8bc10d99aaa8b81.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 18:56:29.016868 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.016841    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:56:29.094735 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.094679    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/997b6acd-c2cf-4208-94a5-87583318be5f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/997b6acd-c2cf-4208-94a5-87583318be5f-cabundle-cert\") pod \"997b6acd-c2cf-4208-94a5-87583318be5f\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") "
Oct 30 18:56:29.094735 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.094727    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/997b6acd-c2cf-4208-94a5-87583318be5f-proxy-tls\" (UniqueName: \"kubernetes.io/secret/997b6acd-c2cf-4208-94a5-87583318be5f-proxy-tls\") pod \"997b6acd-c2cf-4208-94a5-87583318be5f\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") "
Oct 30 18:56:29.094904 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.094753    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997b6acd-c2cf-4208-94a5-87583318be5f-kube-api-access-sg5vd\" (UniqueName: \"kubernetes.io/projected/997b6acd-c2cf-4208-94a5-87583318be5f-kube-api-access-sg5vd\") pod \"997b6acd-c2cf-4208-94a5-87583318be5f\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") "
Oct 30 18:56:29.094904 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.094884    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/997b6acd-c2cf-4208-94a5-87583318be5f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/997b6acd-c2cf-4208-94a5-87583318be5f-kube-rbac-proxy-sar-config\") pod \"997b6acd-c2cf-4208-94a5-87583318be5f\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") "
Oct 30 18:56:29.095011 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.094946    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997b6acd-c2cf-4208-94a5-87583318be5f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/997b6acd-c2cf-4208-94a5-87583318be5f-kserve-provision-location\") pod \"997b6acd-c2cf-4208-94a5-87583318be5f\" (UID: \"997b6acd-c2cf-4208-94a5-87583318be5f\") "
Oct 30 18:56:29.095062 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.095043    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997b6acd-c2cf-4208-94a5-87583318be5f-cabundle-cert" pod "997b6acd-c2cf-4208-94a5-87583318be5f" (UID: "997b6acd-c2cf-4208-94a5-87583318be5f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:29.095280 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.095258    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997b6acd-c2cf-4208-94a5-87583318be5f-kube-rbac-proxy-sar-config" pod "997b6acd-c2cf-4208-94a5-87583318be5f" (UID: "997b6acd-c2cf-4208-94a5-87583318be5f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:29.095280 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.095272    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997b6acd-c2cf-4208-94a5-87583318be5f-kserve-provision-location" pod "997b6acd-c2cf-4208-94a5-87583318be5f" (UID: "997b6acd-c2cf-4208-94a5-87583318be5f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:29.095412 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.095263    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/997b6acd-c2cf-4208-94a5-87583318be5f-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:29.096854 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.096830    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997b6acd-c2cf-4208-94a5-87583318be5f-kube-api-access-sg5vd" pod "997b6acd-c2cf-4208-94a5-87583318be5f" (UID: "997b6acd-c2cf-4208-94a5-87583318be5f"). InnerVolumeSpecName "kube-api-access-sg5vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:29.096960 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.096946    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/997b6acd-c2cf-4208-94a5-87583318be5f-proxy-tls" pod "997b6acd-c2cf-4208-94a5-87583318be5f" (UID: "997b6acd-c2cf-4208-94a5-87583318be5f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:29.196186 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.196158    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/997b6acd-c2cf-4208-94a5-87583318be5f-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:29.196186 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.196183    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg5vd\" (UniqueName: \"kubernetes.io/projected/997b6acd-c2cf-4208-94a5-87583318be5f-kube-api-access-sg5vd\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:29.196342 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.196194    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/997b6acd-c2cf-4208-94a5-87583318be5f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:29.196342 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.196203    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/997b6acd-c2cf-4208-94a5-87583318be5f-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:29.471544 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.471467    4511 generic.go:356] "Generic (PLEG): container finished" podID="997b6acd-c2cf-4208-94a5-87583318be5f" containerID="a1cfe3a9bfc5f1e9cc79b7ccfb202f311b8ef6df12548ea9d8bc10d99aaa8b81" exitCode=0
Oct 30 18:56:29.471544 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.471510    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" event={"ID":"997b6acd-c2cf-4208-94a5-87583318be5f","Type":"ContainerDied","Data":"a1cfe3a9bfc5f1e9cc79b7ccfb202f311b8ef6df12548ea9d8bc10d99aaa8b81"}
Oct 30 18:56:29.471544 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.471527    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"
Oct 30 18:56:29.471940 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.471573    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2" event={"ID":"997b6acd-c2cf-4208-94a5-87583318be5f","Type":"ContainerDied","Data":"3a26fba002d5ce654a35bbfe106f71ef27c2d64c75ccca04cb0d794934f03130"}
Oct 30 18:56:29.471940 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.471609    4511 scope.go:122] "RemoveContainer" containerID="c682d1f5f0581ba3fb78d86e570d94d8a0c1689a785e87ef14f465ede018d599"
Oct 30 18:56:29.480027 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.480009    4511 scope.go:122] "RemoveContainer" containerID="a1cfe3a9bfc5f1e9cc79b7ccfb202f311b8ef6df12548ea9d8bc10d99aaa8b81"
Oct 30 18:56:29.487342 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.487321    4511 scope.go:122] "RemoveContainer" containerID="c4dcbf90cb29af9e6fadd49583682165b50fb01fb6a5f3ebbbd1a426f4640b36"
Oct 30 18:56:29.494562 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.494545    4511 scope.go:122] "RemoveContainer" containerID="c682d1f5f0581ba3fb78d86e570d94d8a0c1689a785e87ef14f465ede018d599"
Oct 30 18:56:29.494825 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:56:29.494806    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c682d1f5f0581ba3fb78d86e570d94d8a0c1689a785e87ef14f465ede018d599\": container with ID starting with c682d1f5f0581ba3fb78d86e570d94d8a0c1689a785e87ef14f465ede018d599 not found: ID does not exist" containerID="c682d1f5f0581ba3fb78d86e570d94d8a0c1689a785e87ef14f465ede018d599"
Oct 30 18:56:29.494895 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.494838    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c682d1f5f0581ba3fb78d86e570d94d8a0c1689a785e87ef14f465ede018d599"} err="failed to get container status \"c682d1f5f0581ba3fb78d86e570d94d8a0c1689a785e87ef14f465ede018d599\": rpc error: code = NotFound desc = could not find container \"c682d1f5f0581ba3fb78d86e570d94d8a0c1689a785e87ef14f465ede018d599\": container with ID starting with c682d1f5f0581ba3fb78d86e570d94d8a0c1689a785e87ef14f465ede018d599 not found: ID does not exist"
Oct 30 18:56:29.494895 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.494863    4511 scope.go:122] "RemoveContainer" containerID="a1cfe3a9bfc5f1e9cc79b7ccfb202f311b8ef6df12548ea9d8bc10d99aaa8b81"
Oct 30 18:56:29.495118 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:56:29.495091    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1cfe3a9bfc5f1e9cc79b7ccfb202f311b8ef6df12548ea9d8bc10d99aaa8b81\": container with ID starting with a1cfe3a9bfc5f1e9cc79b7ccfb202f311b8ef6df12548ea9d8bc10d99aaa8b81 not found: ID does not exist" containerID="a1cfe3a9bfc5f1e9cc79b7ccfb202f311b8ef6df12548ea9d8bc10d99aaa8b81"
Oct 30 18:56:29.495183 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.495140    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1cfe3a9bfc5f1e9cc79b7ccfb202f311b8ef6df12548ea9d8bc10d99aaa8b81"} err="failed to get container status \"a1cfe3a9bfc5f1e9cc79b7ccfb202f311b8ef6df12548ea9d8bc10d99aaa8b81\": rpc error: code = NotFound desc = could not find container \"a1cfe3a9bfc5f1e9cc79b7ccfb202f311b8ef6df12548ea9d8bc10d99aaa8b81\": container with ID starting with a1cfe3a9bfc5f1e9cc79b7ccfb202f311b8ef6df12548ea9d8bc10d99aaa8b81 not found: ID does not exist"
Oct 30 18:56:29.495183 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.495165    4511 scope.go:122] "RemoveContainer" containerID="c4dcbf90cb29af9e6fadd49583682165b50fb01fb6a5f3ebbbd1a426f4640b36"
Oct 30 18:56:29.495453 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:56:29.495431    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4dcbf90cb29af9e6fadd49583682165b50fb01fb6a5f3ebbbd1a426f4640b36\": container with ID starting with c4dcbf90cb29af9e6fadd49583682165b50fb01fb6a5f3ebbbd1a426f4640b36 not found: ID does not exist" containerID="c4dcbf90cb29af9e6fadd49583682165b50fb01fb6a5f3ebbbd1a426f4640b36"
Oct 30 18:56:29.495534 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.495466    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4dcbf90cb29af9e6fadd49583682165b50fb01fb6a5f3ebbbd1a426f4640b36"} err="failed to get container status \"c4dcbf90cb29af9e6fadd49583682165b50fb01fb6a5f3ebbbd1a426f4640b36\": rpc error: code = NotFound desc = could not find container \"c4dcbf90cb29af9e6fadd49583682165b50fb01fb6a5f3ebbbd1a426f4640b36\": container with ID starting with c4dcbf90cb29af9e6fadd49583682165b50fb01fb6a5f3ebbbd1a426f4640b36 not found: ID does not exist"
Oct 30 18:56:29.630238 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.630217    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"]
Oct 30 18:56:29.677422 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:29.677399    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-sklearn-predictor-84c748568-pgwb2"]
Oct 30 18:56:31.655443 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:31.655411    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" path="/var/lib/kubelet/pods/997b6acd-c2cf-4208-94a5-87583318be5f/volumes"
Oct 30 18:56:33.022951 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.022920    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"]
Oct 30 18:56:33.023339 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.023215    4511 cpu_manager.go:414] "containerMap: removing container" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="storage-initializer"
Oct 30 18:56:33.023339 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.023226    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="storage-initializer"
Oct 30 18:56:33.023339 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.023241    4511 cpu_manager.go:414] "containerMap: removing container" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="kube-rbac-proxy"
Oct 30 18:56:33.023339 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.023247    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="kube-rbac-proxy"
Oct 30 18:56:33.023339 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.023265    4511 cpu_manager.go:414] "containerMap: removing container" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="kserve-container"
Oct 30 18:56:33.023339 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.023270    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="kserve-container"
Oct 30 18:56:33.023339 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.023320    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="kserve-container"
Oct 30 18:56:33.023339 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.023330    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="997b6acd-c2cf-4208-94a5-87583318be5f" containerName="kube-rbac-proxy"
Oct 30 18:56:33.026042 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.026024    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:33.075969 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.075942    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 18:56:33.076087 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.076036    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 18:56:33.076087 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.076081    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Oct 30 18:56:33.076212 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.076119    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 18:56:33.123176 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.123151    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 18:56:33.123464 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.123443    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Oct 30 18:56:33.123547 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.123533    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 18:56:33.124721 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.124700    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"]
Oct 30 18:56:33.125868 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.125845    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d559c223-c935-45e6-b924-6b9624206f08-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:33.126014 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.125883    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d559c223-c935-45e6-b924-6b9624206f08-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:33.126014 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.125920    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d559c223-c935-45e6-b924-6b9624206f08-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:33.126104 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.126026    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d559c223-c935-45e6-b924-6b9624206f08-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:33.126104 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.126062    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvqtv\" (UniqueName: \"kubernetes.io/projected/d559c223-c935-45e6-b924-6b9624206f08-kube-api-access-lvqtv\") pod \"isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:33.226921 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.226888    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d559c223-c935-45e6-b924-6b9624206f08-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:33.227097 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.226927    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d559c223-c935-45e6-b924-6b9624206f08-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:33.227097 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.226944    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvqtv\" (UniqueName: \"kubernetes.io/projected/d559c223-c935-45e6-b924-6b9624206f08-kube-api-access-lvqtv\") pod \"isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:33.227097 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.226996    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d559c223-c935-45e6-b924-6b9624206f08-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:33.227097 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.227020    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d559c223-c935-45e6-b924-6b9624206f08-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:33.227418 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.227388    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d559c223-c935-45e6-b924-6b9624206f08-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:33.227685 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.227663    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d559c223-c935-45e6-b924-6b9624206f08-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:33.227723 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.227698    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d559c223-c935-45e6-b924-6b9624206f08-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:33.229540 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.229524    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d559c223-c935-45e6-b924-6b9624206f08-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:33.285916 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.285856    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvqtv\" (UniqueName: \"kubernetes.io/projected/d559c223-c935-45e6-b924-6b9624206f08-kube-api-access-lvqtv\") pod \"isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:33.339932 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.339897    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:33.759853 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:33.759824    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"]
Oct 30 18:56:33.767654 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:56:33.767629    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd559c223_c935_45e6_b924_6b9624206f08.slice/crio-f939a468549f17b6591a519b58f940ed7e355e776fe5ce63d9c58b6da91132b1 WatchSource:0}: Error finding container f939a468549f17b6591a519b58f940ed7e355e776fe5ce63d9c58b6da91132b1: Status 404 returned error can't find the container with id f939a468549f17b6591a519b58f940ed7e355e776fe5ce63d9c58b6da91132b1
Oct 30 18:56:34.491890 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:34.491857    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" event={"ID":"d559c223-c935-45e6-b924-6b9624206f08","Type":"ContainerStarted","Data":"3be6d3907bc64d55e29ff9c53ec8d35fa5f56cb11c348ada63e65944fa639164"}
Oct 30 18:56:34.492310 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:34.491914    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" event={"ID":"d559c223-c935-45e6-b924-6b9624206f08","Type":"ContainerStarted","Data":"f939a468549f17b6591a519b58f940ed7e355e776fe5ce63d9c58b6da91132b1"}
Oct 30 18:56:38.508742 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:38.508711    4511 generic.go:356] "Generic (PLEG): container finished" podID="d559c223-c935-45e6-b924-6b9624206f08" containerID="3be6d3907bc64d55e29ff9c53ec8d35fa5f56cb11c348ada63e65944fa639164" exitCode=0
Oct 30 18:56:38.509116 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:38.508764    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" event={"ID":"d559c223-c935-45e6-b924-6b9624206f08","Type":"ContainerDied","Data":"3be6d3907bc64d55e29ff9c53ec8d35fa5f56cb11c348ada63e65944fa639164"}
Oct 30 18:56:39.515701 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:39.515668    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" event={"ID":"d559c223-c935-45e6-b924-6b9624206f08","Type":"ContainerStarted","Data":"14161160b7f1b0f67fe5816679a15b947039bd0485438ee9a49a3d734eee8e74"}
Oct 30 18:56:39.516227 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:39.515753    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" event={"ID":"d559c223-c935-45e6-b924-6b9624206f08","Type":"ContainerStarted","Data":"a56adc40aab480baa37ffe80418c6df5ed0422908f0643d86115cc731c7658b9"}
Oct 30 18:56:39.516227 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:39.516076    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:39.516227 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:39.516164    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:39.517804 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:39.517776    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Oct 30 18:56:39.632913 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:39.632867    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" podStartSLOduration=3.626310925 podStartE2EDuration="7.632855279s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.006544354s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:39.632157825 +0000 UTC m=+1094.149842724" watchObservedRunningTime="2026-10-30 18:56:39.632855279 +0000 UTC m=+1094.150540157"
Oct 30 18:56:40.156816 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:56:40.156783    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:56:40.520692 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:40.520609    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Oct 30 18:56:45.524804 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:45.524779    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:56:45.525436 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:45.525410    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Oct 30 18:56:55.526323 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:55.526279    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Oct 30 18:56:58.590751 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:58.590722    4511 generic.go:356] "Generic (PLEG): container finished" podID="c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c" containerID="f65a05038656676e5ace29ab4b4259e4ab9b34ae0bdeb1cfe2e3779f9d774423" exitCode=0
Oct 30 18:56:58.591175 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:58.590797    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-54kmx-zspq4" event={"ID":"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c","Type":"ContainerDied","Data":"f65a05038656676e5ace29ab4b4259e4ab9b34ae0bdeb1cfe2e3779f9d774423"}
Oct 30 18:56:59.721375 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:59.721345    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-54kmx-zspq4"
Oct 30 18:56:59.829609 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:59.829576    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-kube-api-access-hvqfz\" (UniqueName: \"kubernetes.io/projected/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-kube-api-access-hvqfz\") pod \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\" (UID: \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\") "
Oct 30 18:56:59.829787 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:59.829621    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-service-ca-bundle\") pod \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\" (UID: \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\") "
Oct 30 18:56:59.829787 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:59.829656    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-trusted-ca-bundle\") pod \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\" (UID: \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\") "
Oct 30 18:56:59.829787 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:59.829708    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-archives-path\") pod \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\" (UID: \"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c\") "
Oct 30 18:56:59.832174 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:59.830392    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-trusted-ca-bundle" pod "c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c" (UID: "c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:59.832174 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:59.830410    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-service-ca-bundle" pod "c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c" (UID: "c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:59.832174 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:59.830443    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-archives-path" pod "c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c" (UID: "c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:59.839021 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:59.838998    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-kube-api-access-hvqfz" pod "c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c" (UID: "c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c"). InnerVolumeSpecName "kube-api-access-hvqfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:59.931481 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:59.931410    4511 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-archives-path\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:59.931481 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:59.931438    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvqfz\" (UniqueName: \"kubernetes.io/projected/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-kube-api-access-hvqfz\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:59.931481 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:59.931449    4511 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-service-ca-bundle\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:59.931481 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:56:59.931457    4511 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c-trusted-ca-bundle\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:00.600108 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:00.600079    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-54kmx-zspq4"
Oct 30 18:57:00.600298 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:00.600081    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-54kmx-zspq4" event={"ID":"c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c","Type":"ContainerDied","Data":"7d922f7078ccb2c129fa8ea60d019cde8d9fd2eca48dc1ce86e6e1358340b910"}
Oct 30 18:57:00.600298 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:00.600203    4511 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d922f7078ccb2c129fa8ea60d019cde8d9fd2eca48dc1ce86e6e1358340b910"
Oct 30 18:57:05.525370 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:05.525325    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Oct 30 18:57:10.156830 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:57:10.156740    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:57:15.526244 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:15.526200    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Oct 30 18:57:25.526054 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:25.526012    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Oct 30 18:57:35.525382 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:35.525347    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Oct 30 18:57:40.155188 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:57:40.155149    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:57:45.526087 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:45.526054    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:57:53.688702 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:53.688661    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"]
Oct 30 18:57:53.689188 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:53.688938    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kserve-container" containerID="cri-o://a56adc40aab480baa37ffe80418c6df5ed0422908f0643d86115cc731c7658b9" gracePeriod=30
Oct 30 18:57:53.689188 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:53.688979    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kube-rbac-proxy" containerID="cri-o://14161160b7f1b0f67fe5816679a15b947039bd0485438ee9a49a3d734eee8e74" gracePeriod=30
Oct 30 18:57:53.804352 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:53.804317    4511 generic.go:356] "Generic (PLEG): container finished" podID="d559c223-c935-45e6-b924-6b9624206f08" containerID="14161160b7f1b0f67fe5816679a15b947039bd0485438ee9a49a3d734eee8e74" exitCode=0
Oct 30 18:57:53.804526 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:53.804487    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" event={"ID":"d559c223-c935-45e6-b924-6b9624206f08","Type":"ContainerDied","Data":"14161160b7f1b0f67fe5816679a15b947039bd0485438ee9a49a3d734eee8e74"}
Oct 30 18:57:55.521147 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:55.521088    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Oct 30 18:57:55.525329 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:55.525303    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Oct 30 18:57:57.728713 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.728689    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:57:57.757664 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.757627    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d559c223-c935-45e6-b924-6b9624206f08-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d559c223-c935-45e6-b924-6b9624206f08-kserve-provision-location\") pod \"d559c223-c935-45e6-b924-6b9624206f08\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") "
Oct 30 18:57:57.757793 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.757709    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d559c223-c935-45e6-b924-6b9624206f08-proxy-tls\" (UniqueName: \"kubernetes.io/secret/d559c223-c935-45e6-b924-6b9624206f08-proxy-tls\") pod \"d559c223-c935-45e6-b924-6b9624206f08\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") "
Oct 30 18:57:57.757793 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.757755    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d559c223-c935-45e6-b924-6b9624206f08-kube-api-access-lvqtv\" (UniqueName: \"kubernetes.io/projected/d559c223-c935-45e6-b924-6b9624206f08-kube-api-access-lvqtv\") pod \"d559c223-c935-45e6-b924-6b9624206f08\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") "
Oct 30 18:57:57.757793 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.757783    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d559c223-c935-45e6-b924-6b9624206f08-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d559c223-c935-45e6-b924-6b9624206f08-cabundle-cert\") pod \"d559c223-c935-45e6-b924-6b9624206f08\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") "
Oct 30 18:57:57.757955 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.757841    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d559c223-c935-45e6-b924-6b9624206f08-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d559c223-c935-45e6-b924-6b9624206f08-kube-rbac-proxy-sar-config\") pod \"d559c223-c935-45e6-b924-6b9624206f08\" (UID: \"d559c223-c935-45e6-b924-6b9624206f08\") "
Oct 30 18:57:57.757955 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.757907    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d559c223-c935-45e6-b924-6b9624206f08-kserve-provision-location" pod "d559c223-c935-45e6-b924-6b9624206f08" (UID: "d559c223-c935-45e6-b924-6b9624206f08"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:57.758060 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.758039    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d559c223-c935-45e6-b924-6b9624206f08-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:57.758214 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.758192    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d559c223-c935-45e6-b924-6b9624206f08-cabundle-cert" pod "d559c223-c935-45e6-b924-6b9624206f08" (UID: "d559c223-c935-45e6-b924-6b9624206f08"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:57.758310 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.758240    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d559c223-c935-45e6-b924-6b9624206f08-kube-rbac-proxy-sar-config" pod "d559c223-c935-45e6-b924-6b9624206f08" (UID: "d559c223-c935-45e6-b924-6b9624206f08"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:57.759859 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.759828    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d559c223-c935-45e6-b924-6b9624206f08-kube-api-access-lvqtv" pod "d559c223-c935-45e6-b924-6b9624206f08" (UID: "d559c223-c935-45e6-b924-6b9624206f08"). InnerVolumeSpecName "kube-api-access-lvqtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:57.759859 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.759850    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d559c223-c935-45e6-b924-6b9624206f08-proxy-tls" pod "d559c223-c935-45e6-b924-6b9624206f08" (UID: "d559c223-c935-45e6-b924-6b9624206f08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:57.819738 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.819712    4511 generic.go:356] "Generic (PLEG): container finished" podID="d559c223-c935-45e6-b924-6b9624206f08" containerID="a56adc40aab480baa37ffe80418c6df5ed0422908f0643d86115cc731c7658b9" exitCode=0
Oct 30 18:57:57.819852 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.819765    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"
Oct 30 18:57:57.819852 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.819771    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" event={"ID":"d559c223-c935-45e6-b924-6b9624206f08","Type":"ContainerDied","Data":"a56adc40aab480baa37ffe80418c6df5ed0422908f0643d86115cc731c7658b9"}
Oct 30 18:57:57.819852 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.819830    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk" event={"ID":"d559c223-c935-45e6-b924-6b9624206f08","Type":"ContainerDied","Data":"f939a468549f17b6591a519b58f940ed7e355e776fe5ce63d9c58b6da91132b1"}
Oct 30 18:57:57.819969 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.819857    4511 scope.go:122] "RemoveContainer" containerID="14161160b7f1b0f67fe5816679a15b947039bd0485438ee9a49a3d734eee8e74"
Oct 30 18:57:57.827947 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.827930    4511 scope.go:122] "RemoveContainer" containerID="a56adc40aab480baa37ffe80418c6df5ed0422908f0643d86115cc731c7658b9"
Oct 30 18:57:57.834830 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.834815    4511 scope.go:122] "RemoveContainer" containerID="3be6d3907bc64d55e29ff9c53ec8d35fa5f56cb11c348ada63e65944fa639164"
Oct 30 18:57:57.841974 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.841957    4511 scope.go:122] "RemoveContainer" containerID="14161160b7f1b0f67fe5816679a15b947039bd0485438ee9a49a3d734eee8e74"
Oct 30 18:57:57.842238 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:57:57.842217    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14161160b7f1b0f67fe5816679a15b947039bd0485438ee9a49a3d734eee8e74\": container with ID starting with 14161160b7f1b0f67fe5816679a15b947039bd0485438ee9a49a3d734eee8e74 not found: ID does not exist" containerID="14161160b7f1b0f67fe5816679a15b947039bd0485438ee9a49a3d734eee8e74"
Oct 30 18:57:57.842294 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.842246    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14161160b7f1b0f67fe5816679a15b947039bd0485438ee9a49a3d734eee8e74"} err="failed to get container status \"14161160b7f1b0f67fe5816679a15b947039bd0485438ee9a49a3d734eee8e74\": rpc error: code = NotFound desc = could not find container \"14161160b7f1b0f67fe5816679a15b947039bd0485438ee9a49a3d734eee8e74\": container with ID starting with 14161160b7f1b0f67fe5816679a15b947039bd0485438ee9a49a3d734eee8e74 not found: ID does not exist"
Oct 30 18:57:57.842294 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.842265    4511 scope.go:122] "RemoveContainer" containerID="a56adc40aab480baa37ffe80418c6df5ed0422908f0643d86115cc731c7658b9"
Oct 30 18:57:57.842507 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:57:57.842480    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56adc40aab480baa37ffe80418c6df5ed0422908f0643d86115cc731c7658b9\": container with ID starting with a56adc40aab480baa37ffe80418c6df5ed0422908f0643d86115cc731c7658b9 not found: ID does not exist" containerID="a56adc40aab480baa37ffe80418c6df5ed0422908f0643d86115cc731c7658b9"
Oct 30 18:57:57.842551 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.842515    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56adc40aab480baa37ffe80418c6df5ed0422908f0643d86115cc731c7658b9"} err="failed to get container status \"a56adc40aab480baa37ffe80418c6df5ed0422908f0643d86115cc731c7658b9\": rpc error: code = NotFound desc = could not find container \"a56adc40aab480baa37ffe80418c6df5ed0422908f0643d86115cc731c7658b9\": container with ID starting with a56adc40aab480baa37ffe80418c6df5ed0422908f0643d86115cc731c7658b9 not found: ID does not exist"
Oct 30 18:57:57.842551 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.842533    4511 scope.go:122] "RemoveContainer" containerID="3be6d3907bc64d55e29ff9c53ec8d35fa5f56cb11c348ada63e65944fa639164"
Oct 30 18:57:57.842732 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:57:57.842718    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3be6d3907bc64d55e29ff9c53ec8d35fa5f56cb11c348ada63e65944fa639164\": container with ID starting with 3be6d3907bc64d55e29ff9c53ec8d35fa5f56cb11c348ada63e65944fa639164 not found: ID does not exist" containerID="3be6d3907bc64d55e29ff9c53ec8d35fa5f56cb11c348ada63e65944fa639164"
Oct 30 18:57:57.842770 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.842737    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be6d3907bc64d55e29ff9c53ec8d35fa5f56cb11c348ada63e65944fa639164"} err="failed to get container status \"3be6d3907bc64d55e29ff9c53ec8d35fa5f56cb11c348ada63e65944fa639164\": rpc error: code = NotFound desc = could not find container \"3be6d3907bc64d55e29ff9c53ec8d35fa5f56cb11c348ada63e65944fa639164\": container with ID starting with 3be6d3907bc64d55e29ff9c53ec8d35fa5f56cb11c348ada63e65944fa639164 not found: ID does not exist"
Oct 30 18:57:57.858719 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.858691    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d559c223-c935-45e6-b924-6b9624206f08-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:57.858719 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.858722    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvqtv\" (UniqueName: \"kubernetes.io/projected/d559c223-c935-45e6-b924-6b9624206f08-kube-api-access-lvqtv\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:57.858822 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.858732    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d559c223-c935-45e6-b924-6b9624206f08-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:57.858822 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.858741    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d559c223-c935-45e6-b924-6b9624206f08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:57.997575 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:57.997553    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"]
Oct 30 18:57:58.001997 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:58.001973    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-xgboost-predictor-8484c6bb7f-2hdwk"]
Oct 30 18:57:59.656541 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:57:59.656510    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d559c223-c935-45e6-b924-6b9624206f08" path="/var/lib/kubelet/pods/d559c223-c935-45e6-b924-6b9624206f08/volumes"
Oct 30 18:58:00.671338 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.671304    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"]
Oct 30 18:58:00.671834 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.671725    4511 cpu_manager.go:414] "containerMap: removing container" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kube-rbac-proxy"
Oct 30 18:58:00.671834 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.671742    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kube-rbac-proxy"
Oct 30 18:58:00.671834 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.671757    4511 cpu_manager.go:414] "containerMap: removing container" podUID="c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c" containerName="insights-gathering"
Oct 30 18:58:00.671834 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.671765    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c" containerName="insights-gathering"
Oct 30 18:58:00.671834 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.671781    4511 cpu_manager.go:414] "containerMap: removing container" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="storage-initializer"
Oct 30 18:58:00.671834 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.671787    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="storage-initializer"
Oct 30 18:58:00.671834 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.671803    4511 cpu_manager.go:414] "containerMap: removing container" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kserve-container"
Oct 30 18:58:00.671834 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.671810    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kserve-container"
Oct 30 18:58:00.672278 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.671882    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kserve-container"
Oct 30 18:58:00.672278 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.671898    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="d559c223-c935-45e6-b924-6b9624206f08" containerName="kube-rbac-proxy"
Oct 30 18:58:00.672278 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.671908    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="c065bfdd-83bb-4ea8-b42a-6fd9c18f4d3c" containerName="insights-gathering"
Oct 30 18:58:00.676483 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.676461    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:00.726357 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.726330    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 18:58:00.773731 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.773700    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 18:58:00.773731 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.773700    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Oct 30 18:58:00.774052 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.773761    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 18:58:00.774052 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.773993    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Oct 30 18:58:00.774228 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.774078    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 18:58:00.774228 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.774166    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 18:58:00.775167 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.775145    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"]
Oct 30 18:58:00.777606 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.777582    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:00.777702 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.777633    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:00.777770 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.777732    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:00.777827 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.777774    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8th4\" (UniqueName: \"kubernetes.io/projected/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kube-api-access-j8th4\") pod \"isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:00.777881 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.777832    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:00.878972 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.878940    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:00.879105 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.878985    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:00.879105 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.879017    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8th4\" (UniqueName: \"kubernetes.io/projected/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kube-api-access-j8th4\") pod \"isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:00.879105 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.879054    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:00.879300 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.879106    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:00.879479 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.879457    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:00.879691 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.879674    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:00.879788 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.879771    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:00.881675 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.881652    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:00.937507 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.937457    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8th4\" (UniqueName: \"kubernetes.io/projected/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kube-api-access-j8th4\") pod \"isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:00.994399 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:00.994377    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:01.415422 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:01.415380    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"]
Oct 30 18:58:01.422880 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:58:01.422851    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7886f5c4_8ea5_4f5f_ba00_1f5fa37c2756.slice/crio-1c057a0b92dc10340c3718c88f865d0f009cf03ceb3251dc24d7dbc573042816 WatchSource:0}: Error finding container 1c057a0b92dc10340c3718c88f865d0f009cf03ceb3251dc24d7dbc573042816: Status 404 returned error can't find the container with id 1c057a0b92dc10340c3718c88f865d0f009cf03ceb3251dc24d7dbc573042816
Oct 30 18:58:01.837594 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:01.837549    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b" event={"ID":"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756","Type":"ContainerStarted","Data":"1a109b08422ecd7f779c1931b5d0dde94c0f269f70fb43947050658e52a496d5"}
Oct 30 18:58:01.837975 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:01.837628    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b" event={"ID":"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756","Type":"ContainerStarted","Data":"1c057a0b92dc10340c3718c88f865d0f009cf03ceb3251dc24d7dbc573042816"}
Oct 30 18:58:05.855916 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:05.855878    4511 generic.go:356] "Generic (PLEG): container finished" podID="7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" containerID="1a109b08422ecd7f779c1931b5d0dde94c0f269f70fb43947050658e52a496d5" exitCode=0
Oct 30 18:58:05.856356 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:05.855953    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b" event={"ID":"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756","Type":"ContainerDied","Data":"1a109b08422ecd7f779c1931b5d0dde94c0f269f70fb43947050658e52a496d5"}
Oct 30 18:58:06.862018 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:06.861979    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b" event={"ID":"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756","Type":"ContainerStarted","Data":"54992ff55cf1eadee25747542a7481e1c1576c5cc44850bb6ba1c9964bee2749"}
Oct 30 18:58:06.862425 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:06.862072    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b" event={"ID":"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756","Type":"ContainerStarted","Data":"5d1edb456789bc25d33241816e046de18c00e8cffeb8461dad721c49fe7a47c4"}
Oct 30 18:58:06.862425 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:06.862397    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:06.978156 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:06.978089    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b" podStartSLOduration=3.016062811 podStartE2EDuration="6.978074498s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.962011687s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:06.977527893 +0000 UTC m=+1181.495212804" watchObservedRunningTime="2026-10-30 18:58:06.978074498 +0000 UTC m=+1181.495759397"
Oct 30 18:58:07.867011 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:07.866982    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:07.868380 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:07.868353    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b" podUID="7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Oct 30 18:58:08.870296 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:08.870252    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b" podUID="7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Oct 30 18:58:10.159116 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:58:10.159078    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:58:13.874987 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:13.874957    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:13.875936 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:13.875920    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:20.767761 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:20.767726    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"]
Oct 30 18:58:20.768213 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:20.767965    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b" podUID="7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" containerName="kserve-container" containerID="cri-o://5d1edb456789bc25d33241816e046de18c00e8cffeb8461dad721c49fe7a47c4" gracePeriod=30
Oct 30 18:58:20.768213 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:20.768016    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b" podUID="7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" containerName="kube-rbac-proxy" containerID="cri-o://54992ff55cf1eadee25747542a7481e1c1576c5cc44850bb6ba1c9964bee2749" gracePeriod=30
Oct 30 18:58:20.920088 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:20.920059    4511 generic.go:356] "Generic (PLEG): container finished" podID="7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" containerID="54992ff55cf1eadee25747542a7481e1c1576c5cc44850bb6ba1c9964bee2749" exitCode=0
Oct 30 18:58:20.920294 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:20.920106    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b" event={"ID":"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756","Type":"ContainerDied","Data":"54992ff55cf1eadee25747542a7481e1c1576c5cc44850bb6ba1c9964bee2749"}
Oct 30 18:58:21.410359 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.410334    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:21.441503 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.441468    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-cabundle-cert\") pod \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") "
Oct 30 18:58:21.441647 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.441532    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-proxy-tls\" (UniqueName: \"kubernetes.io/secret/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-proxy-tls\") pod \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") "
Oct 30 18:58:21.441724 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.441674    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kserve-provision-location\") pod \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") "
Oct 30 18:58:21.441785 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.441720    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kube-api-access-j8th4\" (UniqueName: \"kubernetes.io/projected/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kube-api-access-j8th4\") pod \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") "
Oct 30 18:58:21.441785 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.441760    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kube-rbac-proxy-sar-config\") pod \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\" (UID: \"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756\") "
Oct 30 18:58:21.441941 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.441812    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-cabundle-cert" pod "7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" (UID: "7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:21.442009 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.441990    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:21.442051 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.442014    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kserve-provision-location" pod "7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" (UID: "7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:21.442230 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.442207    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kube-rbac-proxy-sar-config" pod "7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" (UID: "7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:21.443814 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.443781    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-proxy-tls" pod "7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" (UID: "7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:21.443921 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.443863    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kube-api-access-j8th4" pod "7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" (UID: "7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756"). InnerVolumeSpecName "kube-api-access-j8th4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:21.543475 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.543450    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:21.543624 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.543477    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:21.543624 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.543488    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8th4\" (UniqueName: \"kubernetes.io/projected/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kube-api-access-j8th4\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:21.543624 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.543497    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:21.926186 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.926092    4511 generic.go:356] "Generic (PLEG): container finished" podID="7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" containerID="5d1edb456789bc25d33241816e046de18c00e8cffeb8461dad721c49fe7a47c4" exitCode=0
Oct 30 18:58:21.926186 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.926150    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b" event={"ID":"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756","Type":"ContainerDied","Data":"5d1edb456789bc25d33241816e046de18c00e8cffeb8461dad721c49fe7a47c4"}
Oct 30 18:58:21.926186 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.926175    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"
Oct 30 18:58:21.926782 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.926215    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b" event={"ID":"7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756","Type":"ContainerDied","Data":"1c057a0b92dc10340c3718c88f865d0f009cf03ceb3251dc24d7dbc573042816"}
Oct 30 18:58:21.926782 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.926242    4511 scope.go:122] "RemoveContainer" containerID="54992ff55cf1eadee25747542a7481e1c1576c5cc44850bb6ba1c9964bee2749"
Oct 30 18:58:21.934309 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.934291    4511 scope.go:122] "RemoveContainer" containerID="5d1edb456789bc25d33241816e046de18c00e8cffeb8461dad721c49fe7a47c4"
Oct 30 18:58:21.941346 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.941327    4511 scope.go:122] "RemoveContainer" containerID="1a109b08422ecd7f779c1931b5d0dde94c0f269f70fb43947050658e52a496d5"
Oct 30 18:58:21.948307 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.948293    4511 scope.go:122] "RemoveContainer" containerID="54992ff55cf1eadee25747542a7481e1c1576c5cc44850bb6ba1c9964bee2749"
Oct 30 18:58:21.948561 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:58:21.948542    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54992ff55cf1eadee25747542a7481e1c1576c5cc44850bb6ba1c9964bee2749\": container with ID starting with 54992ff55cf1eadee25747542a7481e1c1576c5cc44850bb6ba1c9964bee2749 not found: ID does not exist" containerID="54992ff55cf1eadee25747542a7481e1c1576c5cc44850bb6ba1c9964bee2749"
Oct 30 18:58:21.948616 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.948569    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54992ff55cf1eadee25747542a7481e1c1576c5cc44850bb6ba1c9964bee2749"} err="failed to get container status \"54992ff55cf1eadee25747542a7481e1c1576c5cc44850bb6ba1c9964bee2749\": rpc error: code = NotFound desc = could not find container \"54992ff55cf1eadee25747542a7481e1c1576c5cc44850bb6ba1c9964bee2749\": container with ID starting with 54992ff55cf1eadee25747542a7481e1c1576c5cc44850bb6ba1c9964bee2749 not found: ID does not exist"
Oct 30 18:58:21.948616 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.948585    4511 scope.go:122] "RemoveContainer" containerID="5d1edb456789bc25d33241816e046de18c00e8cffeb8461dad721c49fe7a47c4"
Oct 30 18:58:21.948809 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:58:21.948791    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1edb456789bc25d33241816e046de18c00e8cffeb8461dad721c49fe7a47c4\": container with ID starting with 5d1edb456789bc25d33241816e046de18c00e8cffeb8461dad721c49fe7a47c4 not found: ID does not exist" containerID="5d1edb456789bc25d33241816e046de18c00e8cffeb8461dad721c49fe7a47c4"
Oct 30 18:58:21.948851 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.948821    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1edb456789bc25d33241816e046de18c00e8cffeb8461dad721c49fe7a47c4"} err="failed to get container status \"5d1edb456789bc25d33241816e046de18c00e8cffeb8461dad721c49fe7a47c4\": rpc error: code = NotFound desc = could not find container \"5d1edb456789bc25d33241816e046de18c00e8cffeb8461dad721c49fe7a47c4\": container with ID starting with 5d1edb456789bc25d33241816e046de18c00e8cffeb8461dad721c49fe7a47c4 not found: ID does not exist"
Oct 30 18:58:21.948851 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.948837    4511 scope.go:122] "RemoveContainer" containerID="1a109b08422ecd7f779c1931b5d0dde94c0f269f70fb43947050658e52a496d5"
Oct 30 18:58:21.949074 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:58:21.949054    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a109b08422ecd7f779c1931b5d0dde94c0f269f70fb43947050658e52a496d5\": container with ID starting with 1a109b08422ecd7f779c1931b5d0dde94c0f269f70fb43947050658e52a496d5 not found: ID does not exist" containerID="1a109b08422ecd7f779c1931b5d0dde94c0f269f70fb43947050658e52a496d5"
Oct 30 18:58:21.949193 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:21.949082    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a109b08422ecd7f779c1931b5d0dde94c0f269f70fb43947050658e52a496d5"} err="failed to get container status \"1a109b08422ecd7f779c1931b5d0dde94c0f269f70fb43947050658e52a496d5\": rpc error: code = NotFound desc = could not find container \"1a109b08422ecd7f779c1931b5d0dde94c0f269f70fb43947050658e52a496d5\": container with ID starting with 1a109b08422ecd7f779c1931b5d0dde94c0f269f70fb43947050658e52a496d5 not found: ID does not exist"
Oct 30 18:58:22.083575 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:22.083551    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"]
Oct 30 18:58:22.130876 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:22.130855    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-lightgbm-predictor-6f88b8c8b6-j8c9b"]
Oct 30 18:58:23.655895 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:23.655860    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" path="/var/lib/kubelet/pods/7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756/volumes"
Oct 30 18:58:25.723911 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.723815    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 18:58:25.743563 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.728940    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 18:58:25.743563 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.729906    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"]
Oct 30 18:58:25.743563 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.730243    4511 cpu_manager.go:414] "containerMap: removing container" podUID="7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" containerName="kserve-container"
Oct 30 18:58:25.743563 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.730255    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" containerName="kserve-container"
Oct 30 18:58:25.743563 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.730274    4511 cpu_manager.go:414] "containerMap: removing container" podUID="7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" containerName="kube-rbac-proxy"
Oct 30 18:58:25.743563 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.730279    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" containerName="kube-rbac-proxy"
Oct 30 18:58:25.743563 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.730299    4511 cpu_manager.go:414] "containerMap: removing container" podUID="7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" containerName="storage-initializer"
Oct 30 18:58:25.743563 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.730306    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" containerName="storage-initializer"
Oct 30 18:58:25.743563 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.730360    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" containerName="kserve-container"
Oct 30 18:58:25.743563 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.730369    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="7886f5c4-8ea5-4f5f-ba00-1f5fa37c2756" containerName="kube-rbac-proxy"
Oct 30 18:58:25.743563 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.735643    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:25.776300 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.776274    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thvx4\" (UniqueName: \"kubernetes.io/projected/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kube-api-access-thvx4\") pod \"isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:25.776393 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.776320    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:25.776393 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.776343    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:25.776467 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.776432    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:25.776510 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.776484    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:25.785635 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.785617    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 18:58:25.803546 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.803529    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 18:58:25.803641 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.803551    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Oct 30 18:58:25.832821 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.832799    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Oct 30 18:58:25.832902 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.832833    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 18:58:25.832902 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.832837    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 18:58:25.833233 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.833213    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 18:58:25.834629 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.834611    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"]
Oct 30 18:58:25.877079 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.877052    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:25.877233 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.877098    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thvx4\" (UniqueName: \"kubernetes.io/projected/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kube-api-access-thvx4\") pod \"isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:25.877233 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.877155    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:25.877233 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.877176    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:25.877405 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.877250    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:25.877472 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.877460    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:25.877854 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.877832    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:25.877925 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.877837    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:25.879841 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.879824    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:25.936031 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:25.935999    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thvx4\" (UniqueName: \"kubernetes.io/projected/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kube-api-access-thvx4\") pod \"isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") " pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:26.051983 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:26.051955    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:26.474528 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:26.474436    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"]
Oct 30 18:58:26.482141 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:58:26.482096    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4c6677_21a7_4ffe_a08e_7df2dca9f6ed.slice/crio-c9b2a098ce8b36ce0235c38be25d20777a3445496d27071de18dcb9a3c363db3 WatchSource:0}: Error finding container c9b2a098ce8b36ce0235c38be25d20777a3445496d27071de18dcb9a3c363db3: Status 404 returned error can't find the container with id c9b2a098ce8b36ce0235c38be25d20777a3445496d27071de18dcb9a3c363db3
Oct 30 18:58:26.954189 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:26.954147    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2" event={"ID":"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed","Type":"ContainerStarted","Data":"8c9bbe37a00814085a305a5d4451e9c388ae8497d2f8c37eee48ce32c1f4218a"}
Oct 30 18:58:26.954571 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:26.954231    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2" event={"ID":"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed","Type":"ContainerStarted","Data":"c9b2a098ce8b36ce0235c38be25d20777a3445496d27071de18dcb9a3c363db3"}
Oct 30 18:58:30.971638 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:30.971601    4511 generic.go:356] "Generic (PLEG): container finished" podID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerID="8c9bbe37a00814085a305a5d4451e9c388ae8497d2f8c37eee48ce32c1f4218a" exitCode=0
Oct 30 18:58:30.972088 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:30.971680    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2" event={"ID":"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed","Type":"ContainerDied","Data":"8c9bbe37a00814085a305a5d4451e9c388ae8497d2f8c37eee48ce32c1f4218a"}
Oct 30 18:58:31.977081 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:31.977041    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2" event={"ID":"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed","Type":"ContainerStarted","Data":"6799b66481b16d6197f66df39ba94a52590e434895656f62ae6356a56a33cffa"}
Oct 30 18:58:31.977483 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:31.977106    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2" event={"ID":"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed","Type":"ContainerStarted","Data":"a12ca3d722daf7e1559996f11f7ba8ec0511596d3fc13090c5071f3b1f57cb7b"}
Oct 30 18:58:31.977483 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:31.977328    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:32.139498 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:32.139440    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2" podStartSLOduration=3.488789035 podStartE2EDuration="7.139428382s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.650639347s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:32.091710422 +0000 UTC m=+1206.609395322" watchObservedRunningTime="2026-10-30 18:58:32.139428382 +0000 UTC m=+1206.657113282"
Oct 30 18:58:32.981152 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:32.981105    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:38.989850 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:58:38.989772    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:58:40.156463 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:58:40.156426    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:59:08.991289 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:08.991247    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2" podUID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Oct 30 18:59:10.156222 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:59:10.156179    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:59:18.990991 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:18.990953    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2" podUID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Oct 30 18:59:28.990394 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:28.990356    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2" podUID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Oct 30 18:59:39.421672 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:39.421643    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:59:40.156395 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:59:40.156361    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 18:59:46.061538 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:46.061509    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"]
Oct 30 18:59:46.061970 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:46.061730    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2" podUID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerName="kserve-container" containerID="cri-o://a12ca3d722daf7e1559996f11f7ba8ec0511596d3fc13090c5071f3b1f57cb7b" gracePeriod=30
Oct 30 18:59:46.061970 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:46.061760    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2" podUID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerName="kube-rbac-proxy" containerID="cri-o://6799b66481b16d6197f66df39ba94a52590e434895656f62ae6356a56a33cffa" gracePeriod=30
Oct 30 18:59:46.235516 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:46.235482    4511 generic.go:356] "Generic (PLEG): container finished" podID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerID="6799b66481b16d6197f66df39ba94a52590e434895656f62ae6356a56a33cffa" exitCode=0
Oct 30 18:59:46.235629 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:46.235557    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2" event={"ID":"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed","Type":"ContainerDied","Data":"6799b66481b16d6197f66df39ba94a52590e434895656f62ae6356a56a33cffa"}
Oct 30 18:59:48.985640 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:48.985601    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2" podUID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Oct 30 18:59:48.990979 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:48.990956    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2" podUID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Oct 30 18:59:50.100874 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.100850    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:59:50.191306 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.191243    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kube-api-access-thvx4\" (UniqueName: \"kubernetes.io/projected/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kube-api-access-thvx4\") pod \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") "
Oct 30 18:59:50.191306 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.191280    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-cabundle-cert\") pod \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") "
Oct 30 18:59:50.191477 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.191335    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kube-rbac-proxy-sar-config\") pod \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") "
Oct 30 18:59:50.191477 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.191388    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kserve-provision-location\") pod \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") "
Oct 30 18:59:50.191477 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.191422    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-proxy-tls\") pod \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\" (UID: \"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed\") "
Oct 30 18:59:50.191685 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.191665    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-cabundle-cert" pod "ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" (UID: "ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:50.191743 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.191696    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kserve-provision-location" pod "ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" (UID: "ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:50.191797 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.191749    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kube-rbac-proxy-sar-config" pod "ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" (UID: "ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:50.193398 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.193376    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kube-api-access-thvx4" pod "ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" (UID: "ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed"). InnerVolumeSpecName "kube-api-access-thvx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:50.193509 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.193488    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-proxy-tls" pod "ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" (UID: "ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:50.251689 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.251668    4511 generic.go:356] "Generic (PLEG): container finished" podID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerID="a12ca3d722daf7e1559996f11f7ba8ec0511596d3fc13090c5071f3b1f57cb7b" exitCode=0
Oct 30 18:59:50.251787 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.251720    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"
Oct 30 18:59:50.251787 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.251760    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2" event={"ID":"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed","Type":"ContainerDied","Data":"a12ca3d722daf7e1559996f11f7ba8ec0511596d3fc13090c5071f3b1f57cb7b"}
Oct 30 18:59:50.251854 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.251816    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2" event={"ID":"ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed","Type":"ContainerDied","Data":"c9b2a098ce8b36ce0235c38be25d20777a3445496d27071de18dcb9a3c363db3"}
Oct 30 18:59:50.251854 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.251845    4511 scope.go:122] "RemoveContainer" containerID="6799b66481b16d6197f66df39ba94a52590e434895656f62ae6356a56a33cffa"
Oct 30 18:59:50.259875 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.259862    4511 scope.go:122] "RemoveContainer" containerID="a12ca3d722daf7e1559996f11f7ba8ec0511596d3fc13090c5071f3b1f57cb7b"
Oct 30 18:59:50.272984 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.272948    4511 scope.go:122] "RemoveContainer" containerID="8c9bbe37a00814085a305a5d4451e9c388ae8497d2f8c37eee48ce32c1f4218a"
Oct 30 18:59:50.280959 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.280925    4511 scope.go:122] "RemoveContainer" containerID="6799b66481b16d6197f66df39ba94a52590e434895656f62ae6356a56a33cffa"
Oct 30 18:59:50.281271 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:59:50.281248    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6799b66481b16d6197f66df39ba94a52590e434895656f62ae6356a56a33cffa\": container with ID starting with 6799b66481b16d6197f66df39ba94a52590e434895656f62ae6356a56a33cffa not found: ID does not exist" containerID="6799b66481b16d6197f66df39ba94a52590e434895656f62ae6356a56a33cffa"
Oct 30 18:59:50.281365 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.281281    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6799b66481b16d6197f66df39ba94a52590e434895656f62ae6356a56a33cffa"} err="failed to get container status \"6799b66481b16d6197f66df39ba94a52590e434895656f62ae6356a56a33cffa\": rpc error: code = NotFound desc = could not find container \"6799b66481b16d6197f66df39ba94a52590e434895656f62ae6356a56a33cffa\": container with ID starting with 6799b66481b16d6197f66df39ba94a52590e434895656f62ae6356a56a33cffa not found: ID does not exist"
Oct 30 18:59:50.281365 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.281312    4511 scope.go:122] "RemoveContainer" containerID="a12ca3d722daf7e1559996f11f7ba8ec0511596d3fc13090c5071f3b1f57cb7b"
Oct 30 18:59:50.281626 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:59:50.281609    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a12ca3d722daf7e1559996f11f7ba8ec0511596d3fc13090c5071f3b1f57cb7b\": container with ID starting with a12ca3d722daf7e1559996f11f7ba8ec0511596d3fc13090c5071f3b1f57cb7b not found: ID does not exist" containerID="a12ca3d722daf7e1559996f11f7ba8ec0511596d3fc13090c5071f3b1f57cb7b"
Oct 30 18:59:50.281675 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.281633    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a12ca3d722daf7e1559996f11f7ba8ec0511596d3fc13090c5071f3b1f57cb7b"} err="failed to get container status \"a12ca3d722daf7e1559996f11f7ba8ec0511596d3fc13090c5071f3b1f57cb7b\": rpc error: code = NotFound desc = could not find container \"a12ca3d722daf7e1559996f11f7ba8ec0511596d3fc13090c5071f3b1f57cb7b\": container with ID starting with a12ca3d722daf7e1559996f11f7ba8ec0511596d3fc13090c5071f3b1f57cb7b not found: ID does not exist"
Oct 30 18:59:50.281675 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.281650    4511 scope.go:122] "RemoveContainer" containerID="8c9bbe37a00814085a305a5d4451e9c388ae8497d2f8c37eee48ce32c1f4218a"
Oct 30 18:59:50.281909 ip-10-0-135-66 kubenswrapper[4511]: E1030 18:59:50.281886    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9bbe37a00814085a305a5d4451e9c388ae8497d2f8c37eee48ce32c1f4218a\": container with ID starting with 8c9bbe37a00814085a305a5d4451e9c388ae8497d2f8c37eee48ce32c1f4218a not found: ID does not exist" containerID="8c9bbe37a00814085a305a5d4451e9c388ae8497d2f8c37eee48ce32c1f4218a"
Oct 30 18:59:50.281956 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.281918    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9bbe37a00814085a305a5d4451e9c388ae8497d2f8c37eee48ce32c1f4218a"} err="failed to get container status \"8c9bbe37a00814085a305a5d4451e9c388ae8497d2f8c37eee48ce32c1f4218a\": rpc error: code = NotFound desc = could not find container \"8c9bbe37a00814085a305a5d4451e9c388ae8497d2f8c37eee48ce32c1f4218a\": container with ID starting with 8c9bbe37a00814085a305a5d4451e9c388ae8497d2f8c37eee48ce32c1f4218a not found: ID does not exist"
Oct 30 18:59:50.292253 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.292236    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thvx4\" (UniqueName: \"kubernetes.io/projected/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kube-api-access-thvx4\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:50.292321 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.292256    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:50.292321 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.292265    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:50.292321 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.292274    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:50.292321 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.292283    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:50.371032 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.371011    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"]
Oct 30 18:59:50.375680 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:50.375659    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-sklearn-v2-predictor-5997c57db6-n2lm2"]
Oct 30 18:59:51.655371 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:51.655341    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" path="/var/lib/kubelet/pods/ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed/volumes"
Oct 30 18:59:52.973697 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:52.973664    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"]
Oct 30 18:59:52.974062 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:52.973934    4511 cpu_manager.go:414] "containerMap: removing container" podUID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerName="storage-initializer"
Oct 30 18:59:52.974062 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:52.973944    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerName="storage-initializer"
Oct 30 18:59:52.974062 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:52.973966    4511 cpu_manager.go:414] "containerMap: removing container" podUID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerName="kserve-container"
Oct 30 18:59:52.974062 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:52.973971    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerName="kserve-container"
Oct 30 18:59:52.974062 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:52.973984    4511 cpu_manager.go:414] "containerMap: removing container" podUID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerName="kube-rbac-proxy"
Oct 30 18:59:52.974062 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:52.973989    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerName="kube-rbac-proxy"
Oct 30 18:59:52.974062 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:52.974031    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerName="kserve-container"
Oct 30 18:59:52.974062 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:52.974040    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4c6677-21a7-4ffe-a08e-7df2dca9f6ed" containerName="kube-rbac-proxy"
Oct 30 18:59:52.979327 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:52.979307    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:53.010055 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.010028    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5msj\" (UniqueName: \"kubernetes.io/projected/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kube-api-access-h5msj\") pod \"isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:53.010190 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.010058    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:53.010190 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.010078    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:53.010190 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.010099    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:53.010302 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.010281    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:53.029455 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.029429    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 18:59:53.029552 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.029464    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 18:59:53.076631 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.076607    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 18:59:53.076734 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.076644    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Oct 30 18:59:53.076734 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.076677    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Oct 30 18:59:53.076734 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.076691    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 18:59:53.076734 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.076708    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 18:59:53.079863 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.079842    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"]
Oct 30 18:59:53.111280 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.111254    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5msj\" (UniqueName: \"kubernetes.io/projected/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kube-api-access-h5msj\") pod \"isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:53.111371 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.111284    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:53.111371 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.111304    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:53.111452 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.111423    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:53.111503 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.111459    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:53.111748 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.111728    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:53.111936 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.111916    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:53.112033 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.112017    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:53.114029 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.114010    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:53.169991 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.169970    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5msj\" (UniqueName: \"kubernetes.io/projected/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kube-api-access-h5msj\") pod \"isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") " pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:53.296314 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.296286    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:53.715454 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:53.715381    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"]
Oct 30 18:59:53.722495 ip-10-0-135-66 kubenswrapper[4511]: W1030 18:59:53.722472    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f1616b_f101_4a10_b2a1_a5b8dbb1e280.slice/crio-7610c8d84a879137135284fa8eba49859ede7adf7144d3f9ad1a973eb5bd5edf WatchSource:0}: Error finding container 7610c8d84a879137135284fa8eba49859ede7adf7144d3f9ad1a973eb5bd5edf: Status 404 returned error can't find the container with id 7610c8d84a879137135284fa8eba49859ede7adf7144d3f9ad1a973eb5bd5edf
Oct 30 18:59:54.267959 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:54.267927    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7" event={"ID":"79f1616b-f101-4a10-b2a1-a5b8dbb1e280","Type":"ContainerStarted","Data":"67c830582dbd810a542b62f2139c8fe0f82904601452eba3a363aea093ad751f"}
Oct 30 18:59:54.268341 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:54.267981    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7" event={"ID":"79f1616b-f101-4a10-b2a1-a5b8dbb1e280","Type":"ContainerStarted","Data":"7610c8d84a879137135284fa8eba49859ede7adf7144d3f9ad1a973eb5bd5edf"}
Oct 30 18:59:58.284061 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:58.284028    4511 generic.go:356] "Generic (PLEG): container finished" podID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" containerID="67c830582dbd810a542b62f2139c8fe0f82904601452eba3a363aea093ad751f" exitCode=0
Oct 30 18:59:58.284546 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:58.284083    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7" event={"ID":"79f1616b-f101-4a10-b2a1-a5b8dbb1e280","Type":"ContainerDied","Data":"67c830582dbd810a542b62f2139c8fe0f82904601452eba3a363aea093ad751f"}
Oct 30 18:59:59.291943 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:59.291908    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7" event={"ID":"79f1616b-f101-4a10-b2a1-a5b8dbb1e280","Type":"ContainerStarted","Data":"9670f5170babbe9e0e5fab38a5665b06992586327dcbd82ea51c8dfbb4bd2652"}
Oct 30 18:59:59.292417 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:59.291976    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7" event={"ID":"79f1616b-f101-4a10-b2a1-a5b8dbb1e280","Type":"ContainerStarted","Data":"b9958361554cdc09acaec09d9fb74f6f2693449416b9f4bf74c255cb2e762091"}
Oct 30 18:59:59.292417 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:59.292194    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 18:59:59.407164 ip-10-0-135-66 kubenswrapper[4511]: I1030 18:59:59.407091    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7" podStartSLOduration=3.780480258 podStartE2EDuration="7.407074721s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.626594463s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:59.406616121 +0000 UTC m=+1293.924301043" watchObservedRunningTime="2026-10-30 18:59:59.407074721 +0000 UTC m=+1293.924759622"
Oct 30 19:00:00.295825 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:00:00.295795    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 19:00:06.304669 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:00:06.304597    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 19:00:10.155322 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:00:10.155282    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:00:36.306180 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:00:36.306118    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7" podUID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Oct 30 19:00:40.156439 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:00:40.156401    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:00:46.305140 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:00:46.305100    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7" podUID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Oct 30 19:00:56.723818 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:00:56.723783    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 19:01:03.123224 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:03.123190    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"]
Oct 30 19:01:03.123691 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:03.123426    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7" podUID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" containerName="kserve-container" containerID="cri-o://b9958361554cdc09acaec09d9fb74f6f2693449416b9f4bf74c255cb2e762091" gracePeriod=30
Oct 30 19:01:03.123691 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:03.123495    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7" podUID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" containerName="kube-rbac-proxy" containerID="cri-o://9670f5170babbe9e0e5fab38a5665b06992586327dcbd82ea51c8dfbb4bd2652" gracePeriod=30
Oct 30 19:01:03.526021 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:03.525935    4511 generic.go:356] "Generic (PLEG): container finished" podID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" containerID="9670f5170babbe9e0e5fab38a5665b06992586327dcbd82ea51c8dfbb4bd2652" exitCode=0
Oct 30 19:01:03.526021 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:03.525981    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7" event={"ID":"79f1616b-f101-4a10-b2a1-a5b8dbb1e280","Type":"ContainerDied","Data":"9670f5170babbe9e0e5fab38a5665b06992586327dcbd82ea51c8dfbb4bd2652"}
Oct 30 19:01:06.299548 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:06.299507    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7" podUID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Oct 30 19:01:06.306015 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:06.305986    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7" podUID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Oct 30 19:01:07.761933 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:07.761897    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 19:01:07.866620 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:07.866535    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kube-api-access-h5msj\" (UniqueName: \"kubernetes.io/projected/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kube-api-access-h5msj\") pod \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") "
Oct 30 19:01:07.866620 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:07.866588    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kube-rbac-proxy-sar-config\") pod \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") "
Oct 30 19:01:07.866800 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:07.866681    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-proxy-tls\" (UniqueName: \"kubernetes.io/secret/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-proxy-tls\") pod \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") "
Oct 30 19:01:07.866842 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:07.866794    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-cabundle-cert\") pod \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") "
Oct 30 19:01:07.866842 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:07.866826    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kserve-provision-location\") pod \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\" (UID: \"79f1616b-f101-4a10-b2a1-a5b8dbb1e280\") "
Oct 30 19:01:07.867050 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:07.867020    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kube-rbac-proxy-sar-config" pod "79f1616b-f101-4a10-b2a1-a5b8dbb1e280" (UID: "79f1616b-f101-4a10-b2a1-a5b8dbb1e280"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:07.867198 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:07.867122    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-cabundle-cert" pod "79f1616b-f101-4a10-b2a1-a5b8dbb1e280" (UID: "79f1616b-f101-4a10-b2a1-a5b8dbb1e280"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:07.867262 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:07.867187    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kserve-provision-location" pod "79f1616b-f101-4a10-b2a1-a5b8dbb1e280" (UID: "79f1616b-f101-4a10-b2a1-a5b8dbb1e280"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:07.868869 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:07.868849    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-proxy-tls" pod "79f1616b-f101-4a10-b2a1-a5b8dbb1e280" (UID: "79f1616b-f101-4a10-b2a1-a5b8dbb1e280"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:07.868869 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:07.868852    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kube-api-access-h5msj" pod "79f1616b-f101-4a10-b2a1-a5b8dbb1e280" (UID: "79f1616b-f101-4a10-b2a1-a5b8dbb1e280"). InnerVolumeSpecName "kube-api-access-h5msj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:07.968143 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:07.968097    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:07.968320 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:07.968123    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:07.968320 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:07.968169    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:07.968320 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:07.968180    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5msj\" (UniqueName: \"kubernetes.io/projected/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kube-api-access-h5msj\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:07.968320 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:07.968190    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79f1616b-f101-4a10-b2a1-a5b8dbb1e280-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:08.545270 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:08.545234    4511 generic.go:356] "Generic (PLEG): container finished" podID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" containerID="b9958361554cdc09acaec09d9fb74f6f2693449416b9f4bf74c255cb2e762091" exitCode=0
Oct 30 19:01:08.545449 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:08.545279    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7" event={"ID":"79f1616b-f101-4a10-b2a1-a5b8dbb1e280","Type":"ContainerDied","Data":"b9958361554cdc09acaec09d9fb74f6f2693449416b9f4bf74c255cb2e762091"}
Oct 30 19:01:08.545449 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:08.545303    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"
Oct 30 19:01:08.545449 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:08.545342    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7" event={"ID":"79f1616b-f101-4a10-b2a1-a5b8dbb1e280","Type":"ContainerDied","Data":"7610c8d84a879137135284fa8eba49859ede7adf7144d3f9ad1a973eb5bd5edf"}
Oct 30 19:01:08.545449 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:08.545380    4511 scope.go:122] "RemoveContainer" containerID="9670f5170babbe9e0e5fab38a5665b06992586327dcbd82ea51c8dfbb4bd2652"
Oct 30 19:01:08.554103 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:08.554082    4511 scope.go:122] "RemoveContainer" containerID="b9958361554cdc09acaec09d9fb74f6f2693449416b9f4bf74c255cb2e762091"
Oct 30 19:01:08.561109 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:08.561092    4511 scope.go:122] "RemoveContainer" containerID="67c830582dbd810a542b62f2139c8fe0f82904601452eba3a363aea093ad751f"
Oct 30 19:01:08.567807 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:08.567785    4511 scope.go:122] "RemoveContainer" containerID="9670f5170babbe9e0e5fab38a5665b06992586327dcbd82ea51c8dfbb4bd2652"
Oct 30 19:01:08.568067 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:01:08.568047    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9670f5170babbe9e0e5fab38a5665b06992586327dcbd82ea51c8dfbb4bd2652\": container with ID starting with 9670f5170babbe9e0e5fab38a5665b06992586327dcbd82ea51c8dfbb4bd2652 not found: ID does not exist" containerID="9670f5170babbe9e0e5fab38a5665b06992586327dcbd82ea51c8dfbb4bd2652"
Oct 30 19:01:08.568119 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:08.568076    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9670f5170babbe9e0e5fab38a5665b06992586327dcbd82ea51c8dfbb4bd2652"} err="failed to get container status \"9670f5170babbe9e0e5fab38a5665b06992586327dcbd82ea51c8dfbb4bd2652\": rpc error: code = NotFound desc = could not find container \"9670f5170babbe9e0e5fab38a5665b06992586327dcbd82ea51c8dfbb4bd2652\": container with ID starting with 9670f5170babbe9e0e5fab38a5665b06992586327dcbd82ea51c8dfbb4bd2652 not found: ID does not exist"
Oct 30 19:01:08.568119 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:08.568098    4511 scope.go:122] "RemoveContainer" containerID="b9958361554cdc09acaec09d9fb74f6f2693449416b9f4bf74c255cb2e762091"
Oct 30 19:01:08.568428 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:01:08.568409    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9958361554cdc09acaec09d9fb74f6f2693449416b9f4bf74c255cb2e762091\": container with ID starting with b9958361554cdc09acaec09d9fb74f6f2693449416b9f4bf74c255cb2e762091 not found: ID does not exist" containerID="b9958361554cdc09acaec09d9fb74f6f2693449416b9f4bf74c255cb2e762091"
Oct 30 19:01:08.568500 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:08.568438    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9958361554cdc09acaec09d9fb74f6f2693449416b9f4bf74c255cb2e762091"} err="failed to get container status \"b9958361554cdc09acaec09d9fb74f6f2693449416b9f4bf74c255cb2e762091\": rpc error: code = NotFound desc = could not find container \"b9958361554cdc09acaec09d9fb74f6f2693449416b9f4bf74c255cb2e762091\": container with ID starting with b9958361554cdc09acaec09d9fb74f6f2693449416b9f4bf74c255cb2e762091 not found: ID does not exist"
Oct 30 19:01:08.568500 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:08.568460    4511 scope.go:122] "RemoveContainer" containerID="67c830582dbd810a542b62f2139c8fe0f82904601452eba3a363aea093ad751f"
Oct 30 19:01:08.568697 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:01:08.568681    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67c830582dbd810a542b62f2139c8fe0f82904601452eba3a363aea093ad751f\": container with ID starting with 67c830582dbd810a542b62f2139c8fe0f82904601452eba3a363aea093ad751f not found: ID does not exist" containerID="67c830582dbd810a542b62f2139c8fe0f82904601452eba3a363aea093ad751f"
Oct 30 19:01:08.568737 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:08.568702    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67c830582dbd810a542b62f2139c8fe0f82904601452eba3a363aea093ad751f"} err="failed to get container status \"67c830582dbd810a542b62f2139c8fe0f82904601452eba3a363aea093ad751f\": rpc error: code = NotFound desc = could not find container \"67c830582dbd810a542b62f2139c8fe0f82904601452eba3a363aea093ad751f\": container with ID starting with 67c830582dbd810a542b62f2139c8fe0f82904601452eba3a363aea093ad751f not found: ID does not exist"
Oct 30 19:01:08.750401 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:08.750367    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"]
Oct 30 19:01:08.797834 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:08.797764    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-xgboost-v2-predictor-568f745545-5hfd7"]
Oct 30 19:01:09.655853 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:09.655813    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" path="/var/lib/kubelet/pods/79f1616b-f101-4a10-b2a1-a5b8dbb1e280/volumes"
Oct 30 19:01:10.156753 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:01:10.156719    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:01:12.169537 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.169507    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"]
Oct 30 19:01:12.169939 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.169793    4511 cpu_manager.go:414] "containerMap: removing container" podUID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" containerName="kube-rbac-proxy"
Oct 30 19:01:12.169939 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.169804    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" containerName="kube-rbac-proxy"
Oct 30 19:01:12.169939 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.169818    4511 cpu_manager.go:414] "containerMap: removing container" podUID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" containerName="kserve-container"
Oct 30 19:01:12.169939 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.169823    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" containerName="kserve-container"
Oct 30 19:01:12.169939 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.169837    4511 cpu_manager.go:414] "containerMap: removing container" podUID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" containerName="storage-initializer"
Oct 30 19:01:12.169939 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.169842    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" containerName="storage-initializer"
Oct 30 19:01:12.169939 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.169886    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" containerName="kserve-container"
Oct 30 19:01:12.169939 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.169897    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f1616b-f101-4a10-b2a1-a5b8dbb1e280" containerName="kube-rbac-proxy"
Oct 30 19:01:12.174500 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.174476    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:12.199824 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.199794    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbt76\" (UniqueName: \"kubernetes.io/projected/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kube-api-access-xbt76\") pod \"isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:12.199952 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.199827    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:12.199952 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.199926    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:12.200083 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.199972    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5c711ae-2bac-4f51-86e6-905d99f56cd5-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:12.200083 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.200014    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5c711ae-2bac-4f51-86e6-905d99f56cd5-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:12.271659 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.271629    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:01:12.271659 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.271636    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Oct 30 19:01:12.271659 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.271659    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:01:12.271912 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.271663    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:01:12.271912 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.271693    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:01:12.271912 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.271697    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:01:12.271912 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.271755    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Oct 30 19:01:12.275768 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.275747    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"]
Oct 30 19:01:12.300734 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.300704    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:12.300873 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.300741    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5c711ae-2bac-4f51-86e6-905d99f56cd5-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:12.300873 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.300765    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5c711ae-2bac-4f51-86e6-905d99f56cd5-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:12.300873 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.300813    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbt76\" (UniqueName: \"kubernetes.io/projected/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kube-api-access-xbt76\") pod \"isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:12.300873 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.300842    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:12.301176 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.301152    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:12.301464 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.301445    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5c711ae-2bac-4f51-86e6-905d99f56cd5-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:12.301572 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.301550    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:12.303276 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.303256    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5c711ae-2bac-4f51-86e6-905d99f56cd5-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:12.359960 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.359935    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbt76\" (UniqueName: \"kubernetes.io/projected/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kube-api-access-xbt76\") pod \"isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") " pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:12.490935 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.490856    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:12.912528 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:12.912495    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"]
Oct 30 19:01:12.919953 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:01:12.919928    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c711ae_2bac_4f51_86e6_905d99f56cd5.slice/crio-ef8d0556919dcd94afd122b8dbb086c13f8865ad711198f44057d5c684422945 WatchSource:0}: Error finding container ef8d0556919dcd94afd122b8dbb086c13f8865ad711198f44057d5c684422945: Status 404 returned error can't find the container with id ef8d0556919dcd94afd122b8dbb086c13f8865ad711198f44057d5c684422945
Oct 30 19:01:13.567423 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:13.567386    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx" event={"ID":"c5c711ae-2bac-4f51-86e6-905d99f56cd5","Type":"ContainerStarted","Data":"e0e47e97de9d84d1872253d26ffe5212042e2d261ee596006fbd6afa6fb21031"}
Oct 30 19:01:13.567902 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:13.567443    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx" event={"ID":"c5c711ae-2bac-4f51-86e6-905d99f56cd5","Type":"ContainerStarted","Data":"ef8d0556919dcd94afd122b8dbb086c13f8865ad711198f44057d5c684422945"}
Oct 30 19:01:17.586201 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:17.586168    4511 generic.go:356] "Generic (PLEG): container finished" podID="c5c711ae-2bac-4f51-86e6-905d99f56cd5" containerID="e0e47e97de9d84d1872253d26ffe5212042e2d261ee596006fbd6afa6fb21031" exitCode=0
Oct 30 19:01:17.586614 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:17.586242    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx" event={"ID":"c5c711ae-2bac-4f51-86e6-905d99f56cd5","Type":"ContainerDied","Data":"e0e47e97de9d84d1872253d26ffe5212042e2d261ee596006fbd6afa6fb21031"}
Oct 30 19:01:18.592329 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:18.592293    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx" event={"ID":"c5c711ae-2bac-4f51-86e6-905d99f56cd5","Type":"ContainerStarted","Data":"21f237e4ac5248641052f16f98d5781480ae77dcb7506e2d677b2b31ec98dde1"}
Oct 30 19:01:18.592832 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:18.592353    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx" event={"ID":"c5c711ae-2bac-4f51-86e6-905d99f56cd5","Type":"ContainerStarted","Data":"bcc67692e22037d0b9c473db13a3b154026b18c47156d9b013779ab8c3ada920"}
Oct 30 19:01:18.592832 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:18.592535    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:18.708429 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:18.708376    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx" podStartSLOduration=2.804674944 podStartE2EDuration="6.708364099s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.903689155s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:18.708220413 +0000 UTC m=+1373.225905314" watchObservedRunningTime="2026-10-30 19:01:18.708364099 +0000 UTC m=+1373.226048998"
Oct 30 19:01:19.596737 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:19.596705    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:25.606961 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:25.606936    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:01:40.156160 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:01:40.156053    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:01:55.666355 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:01:55.666328    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:02:02.445995 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:02.445965    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"]
Oct 30 19:02:02.446413 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:02.446238    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx" podUID="c5c711ae-2bac-4f51-86e6-905d99f56cd5" containerName="kserve-container" containerID="cri-o://bcc67692e22037d0b9c473db13a3b154026b18c47156d9b013779ab8c3ada920" gracePeriod=30
Oct 30 19:02:02.446413 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:02.446283    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx" podUID="c5c711ae-2bac-4f51-86e6-905d99f56cd5" containerName="kube-rbac-proxy" containerID="cri-o://21f237e4ac5248641052f16f98d5781480ae77dcb7506e2d677b2b31ec98dde1" gracePeriod=30
Oct 30 19:02:02.750619 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:02.750533    4511 generic.go:356] "Generic (PLEG): container finished" podID="c5c711ae-2bac-4f51-86e6-905d99f56cd5" containerID="21f237e4ac5248641052f16f98d5781480ae77dcb7506e2d677b2b31ec98dde1" exitCode=0
Oct 30 19:02:02.750774 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:02.750614    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx" event={"ID":"c5c711ae-2bac-4f51-86e6-905d99f56cd5","Type":"ContainerDied","Data":"21f237e4ac5248641052f16f98d5781480ae77dcb7506e2d677b2b31ec98dde1"}
Oct 30 19:02:03.085201 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.085174    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:02:03.197815 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.197780    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c711ae-2bac-4f51-86e6-905d99f56cd5-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5c711ae-2bac-4f51-86e6-905d99f56cd5-cabundle-cert\") pod \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") "
Oct 30 19:02:03.197989 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.197841    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5c711ae-2bac-4f51-86e6-905d99f56cd5-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5c711ae-2bac-4f51-86e6-905d99f56cd5-proxy-tls\") pod \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") "
Oct 30 19:02:03.197989 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.197876    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kube-api-access-xbt76\" (UniqueName: \"kubernetes.io/projected/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kube-api-access-xbt76\") pod \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") "
Oct 30 19:02:03.197989 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.197965    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kserve-provision-location\") pod \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") "
Oct 30 19:02:03.198166 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.198016    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kube-rbac-proxy-sar-config\") pod \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\" (UID: \"c5c711ae-2bac-4f51-86e6-905d99f56cd5\") "
Oct 30 19:02:03.198234 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.198200    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c711ae-2bac-4f51-86e6-905d99f56cd5-cabundle-cert" pod "c5c711ae-2bac-4f51-86e6-905d99f56cd5" (UID: "c5c711ae-2bac-4f51-86e6-905d99f56cd5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:03.198426 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.198402    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5c711ae-2bac-4f51-86e6-905d99f56cd5-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:03.198724 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.198429    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kserve-provision-location" pod "c5c711ae-2bac-4f51-86e6-905d99f56cd5" (UID: "c5c711ae-2bac-4f51-86e6-905d99f56cd5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:02:03.198724 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.198457    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kube-rbac-proxy-sar-config" pod "c5c711ae-2bac-4f51-86e6-905d99f56cd5" (UID: "c5c711ae-2bac-4f51-86e6-905d99f56cd5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:03.200236 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.200205    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c711ae-2bac-4f51-86e6-905d99f56cd5-proxy-tls" pod "c5c711ae-2bac-4f51-86e6-905d99f56cd5" (UID: "c5c711ae-2bac-4f51-86e6-905d99f56cd5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:02:03.200444 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.200416    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kube-api-access-xbt76" pod "c5c711ae-2bac-4f51-86e6-905d99f56cd5" (UID: "c5c711ae-2bac-4f51-86e6-905d99f56cd5"). InnerVolumeSpecName "kube-api-access-xbt76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:03.299609 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.299575    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5c711ae-2bac-4f51-86e6-905d99f56cd5-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:03.299609 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.299612    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbt76\" (UniqueName: \"kubernetes.io/projected/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kube-api-access-xbt76\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:03.299791 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.299623    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:03.299791 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.299633    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5c711ae-2bac-4f51-86e6-905d99f56cd5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:03.756489 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.756452    4511 generic.go:356] "Generic (PLEG): container finished" podID="c5c711ae-2bac-4f51-86e6-905d99f56cd5" containerID="bcc67692e22037d0b9c473db13a3b154026b18c47156d9b013779ab8c3ada920" exitCode=0
Oct 30 19:02:03.756868 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.756515    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"
Oct 30 19:02:03.756868 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.756541    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx" event={"ID":"c5c711ae-2bac-4f51-86e6-905d99f56cd5","Type":"ContainerDied","Data":"bcc67692e22037d0b9c473db13a3b154026b18c47156d9b013779ab8c3ada920"}
Oct 30 19:02:03.756868 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.756604    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx" event={"ID":"c5c711ae-2bac-4f51-86e6-905d99f56cd5","Type":"ContainerDied","Data":"ef8d0556919dcd94afd122b8dbb086c13f8865ad711198f44057d5c684422945"}
Oct 30 19:02:03.756868 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.756631    4511 scope.go:122] "RemoveContainer" containerID="21f237e4ac5248641052f16f98d5781480ae77dcb7506e2d677b2b31ec98dde1"
Oct 30 19:02:03.764607 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.764588    4511 scope.go:122] "RemoveContainer" containerID="bcc67692e22037d0b9c473db13a3b154026b18c47156d9b013779ab8c3ada920"
Oct 30 19:02:03.771556 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.771538    4511 scope.go:122] "RemoveContainer" containerID="e0e47e97de9d84d1872253d26ffe5212042e2d261ee596006fbd6afa6fb21031"
Oct 30 19:02:03.778740 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.778719    4511 scope.go:122] "RemoveContainer" containerID="21f237e4ac5248641052f16f98d5781480ae77dcb7506e2d677b2b31ec98dde1"
Oct 30 19:02:03.779021 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:02:03.778999    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f237e4ac5248641052f16f98d5781480ae77dcb7506e2d677b2b31ec98dde1\": container with ID starting with 21f237e4ac5248641052f16f98d5781480ae77dcb7506e2d677b2b31ec98dde1 not found: ID does not exist" containerID="21f237e4ac5248641052f16f98d5781480ae77dcb7506e2d677b2b31ec98dde1"
Oct 30 19:02:03.779100 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.779031    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f237e4ac5248641052f16f98d5781480ae77dcb7506e2d677b2b31ec98dde1"} err="failed to get container status \"21f237e4ac5248641052f16f98d5781480ae77dcb7506e2d677b2b31ec98dde1\": rpc error: code = NotFound desc = could not find container \"21f237e4ac5248641052f16f98d5781480ae77dcb7506e2d677b2b31ec98dde1\": container with ID starting with 21f237e4ac5248641052f16f98d5781480ae77dcb7506e2d677b2b31ec98dde1 not found: ID does not exist"
Oct 30 19:02:03.779100 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.779054    4511 scope.go:122] "RemoveContainer" containerID="bcc67692e22037d0b9c473db13a3b154026b18c47156d9b013779ab8c3ada920"
Oct 30 19:02:03.779337 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:02:03.779318    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc67692e22037d0b9c473db13a3b154026b18c47156d9b013779ab8c3ada920\": container with ID starting with bcc67692e22037d0b9c473db13a3b154026b18c47156d9b013779ab8c3ada920 not found: ID does not exist" containerID="bcc67692e22037d0b9c473db13a3b154026b18c47156d9b013779ab8c3ada920"
Oct 30 19:02:03.779384 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.779345    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc67692e22037d0b9c473db13a3b154026b18c47156d9b013779ab8c3ada920"} err="failed to get container status \"bcc67692e22037d0b9c473db13a3b154026b18c47156d9b013779ab8c3ada920\": rpc error: code = NotFound desc = could not find container \"bcc67692e22037d0b9c473db13a3b154026b18c47156d9b013779ab8c3ada920\": container with ID starting with bcc67692e22037d0b9c473db13a3b154026b18c47156d9b013779ab8c3ada920 not found: ID does not exist"
Oct 30 19:02:03.779384 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.779367    4511 scope.go:122] "RemoveContainer" containerID="e0e47e97de9d84d1872253d26ffe5212042e2d261ee596006fbd6afa6fb21031"
Oct 30 19:02:03.779628 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:02:03.779611    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e47e97de9d84d1872253d26ffe5212042e2d261ee596006fbd6afa6fb21031\": container with ID starting with e0e47e97de9d84d1872253d26ffe5212042e2d261ee596006fbd6afa6fb21031 not found: ID does not exist" containerID="e0e47e97de9d84d1872253d26ffe5212042e2d261ee596006fbd6afa6fb21031"
Oct 30 19:02:03.779682 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.779632    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e47e97de9d84d1872253d26ffe5212042e2d261ee596006fbd6afa6fb21031"} err="failed to get container status \"e0e47e97de9d84d1872253d26ffe5212042e2d261ee596006fbd6afa6fb21031\": rpc error: code = NotFound desc = could not find container \"e0e47e97de9d84d1872253d26ffe5212042e2d261ee596006fbd6afa6fb21031\": container with ID starting with e0e47e97de9d84d1872253d26ffe5212042e2d261ee596006fbd6afa6fb21031 not found: ID does not exist"
Oct 30 19:02:03.914836 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.914757    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"]
Oct 30 19:02:03.962286 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:03.962252    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-predictive-lightgbm-v2-predictor-f8db4887d-r8fbx"]
Oct 30 19:02:05.655927 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:02:05.655896    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c711ae-2bac-4f51-86e6-905d99f56cd5" path="/var/lib/kubelet/pods/c5c711ae-2bac-4f51-86e6-905d99f56cd5/volumes"
Oct 30 19:02:10.156512 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:02:10.156471    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:02:40.156482 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:02:40.156447    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:03:10.156715 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:03:10.156638    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:03:25.744556 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:03:25.744454    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 19:03:25.750800 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:03:25.750782    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 19:03:40.156314 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:03:40.156277    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:04:10.155866 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:04:10.155831    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:04:33.385666 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.385633    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"]
Oct 30 19:04:33.386297 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.386046    4511 cpu_manager.go:414] "containerMap: removing container" podUID="c5c711ae-2bac-4f51-86e6-905d99f56cd5" containerName="kube-rbac-proxy"
Oct 30 19:04:33.386297 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.386062    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c711ae-2bac-4f51-86e6-905d99f56cd5" containerName="kube-rbac-proxy"
Oct 30 19:04:33.386297 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.386095    4511 cpu_manager.go:414] "containerMap: removing container" podUID="c5c711ae-2bac-4f51-86e6-905d99f56cd5" containerName="storage-initializer"
Oct 30 19:04:33.386297 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.386103    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c711ae-2bac-4f51-86e6-905d99f56cd5" containerName="storage-initializer"
Oct 30 19:04:33.386297 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.386122    4511 cpu_manager.go:414] "containerMap: removing container" podUID="c5c711ae-2bac-4f51-86e6-905d99f56cd5" containerName="kserve-container"
Oct 30 19:04:33.386297 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.386153    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c711ae-2bac-4f51-86e6-905d99f56cd5" containerName="kserve-container"
Oct 30 19:04:33.386297 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.386231    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c711ae-2bac-4f51-86e6-905d99f56cd5" containerName="kserve-container"
Oct 30 19:04:33.386297 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.386243    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c711ae-2bac-4f51-86e6-905d99f56cd5" containerName="kube-rbac-proxy"
Oct 30 19:04:33.389308 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.389286    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:04:33.439678 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.439654    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 30 19:04:33.439678 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.439664    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:04:33.473714 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.473692    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-74989457cb-6t7f8\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") " pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:04:33.473826 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.473722    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-74989457cb-6t7f8\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") " pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:04:33.473826 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.473759    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-74989457cb-6t7f8\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") " pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:04:33.473983 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.473957    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-74989457cb-6t7f8\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") " pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:04:33.474044 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.474020    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9hbt\" (UniqueName: \"kubernetes.io/projected/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kube-api-access-l9hbt\") pod \"sklearn-v2-mlserver-predictor-74989457cb-6t7f8\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") " pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:04:33.486983 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.486958    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Oct 30 19:04:33.487085 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.486985    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:04:33.487085 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.487026    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:04:33.487085 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.487026    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:04:33.487085 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.487063    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:04:33.491551 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.491532    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"]
Oct 30 19:04:33.574965 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.574935    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-74989457cb-6t7f8\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") " pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:04:33.575081 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.574986    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-74989457cb-6t7f8\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") " pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:04:33.575081 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.575023    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9hbt\" (UniqueName: \"kubernetes.io/projected/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kube-api-access-l9hbt\") pod \"sklearn-v2-mlserver-predictor-74989457cb-6t7f8\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") " pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:04:33.575081 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.575046    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-74989457cb-6t7f8\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") " pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:04:33.575081 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.575068    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-74989457cb-6t7f8\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") " pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:04:33.575474 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.575446    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-74989457cb-6t7f8\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") " pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:04:33.575676 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.575652    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-74989457cb-6t7f8\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") " pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:04:33.575729 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.575662    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-74989457cb-6t7f8\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") " pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:04:33.577574 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.577558    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-74989457cb-6t7f8\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") " pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:04:33.633609 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.633576    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9hbt\" (UniqueName: \"kubernetes.io/projected/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kube-api-access-l9hbt\") pod \"sklearn-v2-mlserver-predictor-74989457cb-6t7f8\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") " pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:04:33.705373 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:33.705322    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:04:34.127427 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:34.127392    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"]
Oct 30 19:04:34.134892 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:04:34.134861    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcb1ec8_4e7a_4744_8bfa_d663e0272b12.slice/crio-dc4d5e6d9da3e62db4fd2230ab8d25e014456b576c234a21a5af1e4b190a9a74 WatchSource:0}: Error finding container dc4d5e6d9da3e62db4fd2230ab8d25e014456b576c234a21a5af1e4b190a9a74: Status 404 returned error can't find the container with id dc4d5e6d9da3e62db4fd2230ab8d25e014456b576c234a21a5af1e4b190a9a74
Oct 30 19:04:34.284386 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:34.284357    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8" event={"ID":"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12","Type":"ContainerStarted","Data":"2a3eba5b9f07beb816307f524ddf4f457c2ef79e65570cd43de94fbfa01e7284"}
Oct 30 19:04:34.284532 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:34.284453    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8" event={"ID":"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12","Type":"ContainerStarted","Data":"dc4d5e6d9da3e62db4fd2230ab8d25e014456b576c234a21a5af1e4b190a9a74"}
Oct 30 19:04:38.301306 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:38.301274    4511 generic.go:356] "Generic (PLEG): container finished" podID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" containerID="2a3eba5b9f07beb816307f524ddf4f457c2ef79e65570cd43de94fbfa01e7284" exitCode=0
Oct 30 19:04:38.301672 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:38.301347    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8" event={"ID":"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12","Type":"ContainerDied","Data":"2a3eba5b9f07beb816307f524ddf4f457c2ef79e65570cd43de94fbfa01e7284"}
Oct 30 19:04:38.302444 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:04:38.302430    4511 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:04:40.170598 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:04:40.170517    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:05:10.160767 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:05:10.160724    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:05:38.522028 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.521997    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"]
Oct 30 19:05:38.591696 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.591653    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"]
Oct 30 19:05:38.591854 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.591785    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"
Oct 30 19:05:38.642291 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.642261    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:05:38.642435 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.642349    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Oct 30 19:05:38.689725 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.689676    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"message-dumper-predictor-serving-cert\""
Oct 30 19:05:38.689725 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.689709    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:05:38.689725 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.689727    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:05:38.742349 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.742320    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-proxy-tls\") pod \"message-dumper-predictor-6476b87f8-lxmtg\" (UID: \"9a3d5d69-7fab-4807-adda-f5fdc0ff2476\") " pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"
Oct 30 19:05:38.742499 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.742367    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6476b87f8-lxmtg\" (UID: \"9a3d5d69-7fab-4807-adda-f5fdc0ff2476\") " pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"
Oct 30 19:05:38.742682 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.742662    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wqs8\" (UniqueName: \"kubernetes.io/projected/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-kube-api-access-4wqs8\") pod \"message-dumper-predictor-6476b87f8-lxmtg\" (UID: \"9a3d5d69-7fab-4807-adda-f5fdc0ff2476\") " pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"
Oct 30 19:05:38.843990 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.843948    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wqs8\" (UniqueName: \"kubernetes.io/projected/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-kube-api-access-4wqs8\") pod \"message-dumper-predictor-6476b87f8-lxmtg\" (UID: \"9a3d5d69-7fab-4807-adda-f5fdc0ff2476\") " pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"
Oct 30 19:05:38.844895 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.844004    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-proxy-tls\") pod \"message-dumper-predictor-6476b87f8-lxmtg\" (UID: \"9a3d5d69-7fab-4807-adda-f5fdc0ff2476\") " pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"
Oct 30 19:05:38.844895 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.844044    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6476b87f8-lxmtg\" (UID: \"9a3d5d69-7fab-4807-adda-f5fdc0ff2476\") " pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"
Oct 30 19:05:38.844895 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.844849    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6476b87f8-lxmtg\" (UID: \"9a3d5d69-7fab-4807-adda-f5fdc0ff2476\") " pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"
Oct 30 19:05:38.847721 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.847695    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-proxy-tls\") pod \"message-dumper-predictor-6476b87f8-lxmtg\" (UID: \"9a3d5d69-7fab-4807-adda-f5fdc0ff2476\") " pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"
Oct 30 19:05:38.903169 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.903143    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wqs8\" (UniqueName: \"kubernetes.io/projected/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-kube-api-access-4wqs8\") pod \"message-dumper-predictor-6476b87f8-lxmtg\" (UID: \"9a3d5d69-7fab-4807-adda-f5fdc0ff2476\") " pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"
Oct 30 19:05:38.933038 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:38.932834    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"
Oct 30 19:05:39.332100 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:39.332068    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"]
Oct 30 19:05:39.338872 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:05:39.338835    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3d5d69_7fab_4807_adda_f5fdc0ff2476.slice/crio-b2ca3e897a48604560acef28c63f9d66ca4259d8248a452d53f1ad3aff40758e WatchSource:0}: Error finding container b2ca3e897a48604560acef28c63f9d66ca4259d8248a452d53f1ad3aff40758e: Status 404 returned error can't find the container with id b2ca3e897a48604560acef28c63f9d66ca4259d8248a452d53f1ad3aff40758e
Oct 30 19:05:39.568257 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:39.568220    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg" event={"ID":"9a3d5d69-7fab-4807-adda-f5fdc0ff2476","Type":"ContainerStarted","Data":"b2ca3e897a48604560acef28c63f9d66ca4259d8248a452d53f1ad3aff40758e"}
Oct 30 19:05:40.159295 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:05:40.159255    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:05:42.583885 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:42.583847    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg" event={"ID":"9a3d5d69-7fab-4807-adda-f5fdc0ff2476","Type":"ContainerStarted","Data":"fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a"}
Oct 30 19:05:42.584349 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:42.583917    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg" event={"ID":"9a3d5d69-7fab-4807-adda-f5fdc0ff2476","Type":"ContainerStarted","Data":"8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787"}
Oct 30 19:05:42.584349 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:42.583980    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"
Oct 30 19:05:42.696510 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:42.696449    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg" podStartSLOduration=2.323263568 podStartE2EDuration="4.696429877s" totalImagesPullingTime="2.373166309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:42.696277089 +0000 UTC m=+1637.213961994" watchObservedRunningTime="2026-10-30 19:05:42.696429877 +0000 UTC m=+1637.214114779"
Oct 30 19:05:43.588756 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:43.588718    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"
Oct 30 19:05:43.590978 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:43.590943    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"
Oct 30 19:05:50.603253 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:05:50.603224    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"
Oct 30 19:06:10.163570 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:06:10.163478    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:06:36.824195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:06:36.824160    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8" event={"ID":"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12","Type":"ContainerStarted","Data":"71faa11797a77bad1014f5cb0a923f15226f5654df5590a71ea72a2409fc59b5"}
Oct 30 19:06:37.829777 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:06:37.829744    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8" event={"ID":"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12","Type":"ContainerStarted","Data":"d596f3a21ea843e687d5b7d5784d5cc77a0bece3a4a501ae7e95c93d201c0d62"}
Oct 30 19:06:37.830175 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:06:37.829801    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:06:37.830175 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:06:37.829943    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:06:37.996103 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:06:37.996054    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8" podStartSLOduration=2.801096214 podStartE2EDuration="2m4.996039614s" totalImagesPullingTime="1m58.362425657s" totalInitContainerRuntime="3.832517743s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:37.995279635 +0000 UTC m=+1692.512964534" watchObservedRunningTime="2026-10-30 19:06:37.996039614 +0000 UTC m=+1692.513724514"
Oct 30 19:06:40.156748 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:06:40.156715    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:06:43.839074 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:06:43.839046    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:07:10.156401 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:07:10.156357    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:07:13.916961 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:13.916912    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8" podUID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 30 19:07:23.841677 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:23.841650    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:07:33.133155 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:33.133107    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"]
Oct 30 19:07:33.133612 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:33.133365    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8" podUID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" containerName="kserve-container" containerID="cri-o://71faa11797a77bad1014f5cb0a923f15226f5654df5590a71ea72a2409fc59b5" gracePeriod=30
Oct 30 19:07:33.133612 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:33.133406    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8" podUID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" containerName="kube-rbac-proxy" containerID="cri-o://d596f3a21ea843e687d5b7d5784d5cc77a0bece3a4a501ae7e95c93d201c0d62" gracePeriod=30
Oct 30 19:07:33.635886 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:33.635853    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-6eaea25a75_message-dumper-predictor-6476b87f8-lxmtg_9a3d5d69-7fab-4807-adda-f5fdc0ff2476/kserve-container/0.log"
Oct 30 19:07:33.834997 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:33.834960    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8" podUID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Oct 30 19:07:34.050399 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:34.050369    4511 generic.go:356] "Generic (PLEG): container finished" podID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" containerID="d596f3a21ea843e687d5b7d5784d5cc77a0bece3a4a501ae7e95c93d201c0d62" exitCode=0
Oct 30 19:07:34.050572 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:34.050408    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8" event={"ID":"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12","Type":"ContainerDied","Data":"d596f3a21ea843e687d5b7d5784d5cc77a0bece3a4a501ae7e95c93d201c0d62"}
Oct 30 19:07:34.881270 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:34.881227    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8" podUID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.35:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:07:35.461394 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:35.461364    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"]
Oct 30 19:07:35.461641 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:35.461611    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg" podUID="9a3d5d69-7fab-4807-adda-f5fdc0ff2476" containerName="kserve-container" containerID="cri-o://8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787" gracePeriod=30
Oct 30 19:07:35.461776 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:35.461638    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg" podUID="9a3d5d69-7fab-4807-adda-f5fdc0ff2476" containerName="kube-rbac-proxy" containerID="cri-o://fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a" gracePeriod=30
Oct 30 19:07:35.599364 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:35.599310    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg" podUID="9a3d5d69-7fab-4807-adda-f5fdc0ff2476" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Oct 30 19:07:35.694946 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:35.694923    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"
Oct 30 19:07:35.779835 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:35.779776    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-proxy-tls\") pod \"9a3d5d69-7fab-4807-adda-f5fdc0ff2476\" (UID: \"9a3d5d69-7fab-4807-adda-f5fdc0ff2476\") "
Oct 30 19:07:35.779932 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:35.779848    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-kube-api-access-4wqs8\" (UniqueName: \"kubernetes.io/projected/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-kube-api-access-4wqs8\") pod \"9a3d5d69-7fab-4807-adda-f5fdc0ff2476\" (UID: \"9a3d5d69-7fab-4807-adda-f5fdc0ff2476\") "
Oct 30 19:07:35.779932 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:35.779888    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-kube-rbac-proxy-sar-config\") pod \"9a3d5d69-7fab-4807-adda-f5fdc0ff2476\" (UID: \"9a3d5d69-7fab-4807-adda-f5fdc0ff2476\") "
Oct 30 19:07:35.780260 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:35.780234    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-kube-rbac-proxy-sar-config" pod "9a3d5d69-7fab-4807-adda-f5fdc0ff2476" (UID: "9a3d5d69-7fab-4807-adda-f5fdc0ff2476"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:35.782874 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:35.782853    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-proxy-tls" pod "9a3d5d69-7fab-4807-adda-f5fdc0ff2476" (UID: "9a3d5d69-7fab-4807-adda-f5fdc0ff2476"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:35.783069 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:35.783041    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-kube-api-access-4wqs8" pod "9a3d5d69-7fab-4807-adda-f5fdc0ff2476" (UID: "9a3d5d69-7fab-4807-adda-f5fdc0ff2476"). InnerVolumeSpecName "kube-api-access-4wqs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:35.880986 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:35.880960    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:35.881104 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:35.881009    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wqs8\" (UniqueName: \"kubernetes.io/projected/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-kube-api-access-4wqs8\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:35.881104 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:35.881024    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a3d5d69-7fab-4807-adda-f5fdc0ff2476-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:36.059605 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.059576    4511 generic.go:356] "Generic (PLEG): container finished" podID="9a3d5d69-7fab-4807-adda-f5fdc0ff2476" containerID="fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a" exitCode=0
Oct 30 19:07:36.059605 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.059601    4511 generic.go:356] "Generic (PLEG): container finished" podID="9a3d5d69-7fab-4807-adda-f5fdc0ff2476" containerID="8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787" exitCode=2
Oct 30 19:07:36.060072 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.059631    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"
Oct 30 19:07:36.060072 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.059657    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg" event={"ID":"9a3d5d69-7fab-4807-adda-f5fdc0ff2476","Type":"ContainerDied","Data":"fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a"}
Oct 30 19:07:36.060072 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.059730    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg" event={"ID":"9a3d5d69-7fab-4807-adda-f5fdc0ff2476","Type":"ContainerDied","Data":"8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787"}
Oct 30 19:07:36.060072 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.059763    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg" event={"ID":"9a3d5d69-7fab-4807-adda-f5fdc0ff2476","Type":"ContainerDied","Data":"b2ca3e897a48604560acef28c63f9d66ca4259d8248a452d53f1ad3aff40758e"}
Oct 30 19:07:36.060072 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.059795    4511 scope.go:122] "RemoveContainer" containerID="fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a"
Oct 30 19:07:36.068240 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.068224    4511 scope.go:122] "RemoveContainer" containerID="8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787"
Oct 30 19:07:36.075468 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.075453    4511 scope.go:122] "RemoveContainer" containerID="fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a"
Oct 30 19:07:36.075681 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:07:36.075666    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a\": container with ID starting with fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a not found: ID does not exist" containerID="fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a"
Oct 30 19:07:36.075746 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.075689    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a"} err="failed to get container status \"fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a\": rpc error: code = NotFound desc = could not find container \"fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a\": container with ID starting with fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a not found: ID does not exist"
Oct 30 19:07:36.075746 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.075707    4511 scope.go:122] "RemoveContainer" containerID="8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787"
Oct 30 19:07:36.075953 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:07:36.075937    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787\": container with ID starting with 8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787 not found: ID does not exist" containerID="8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787"
Oct 30 19:07:36.075999 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.075957    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787"} err="failed to get container status \"8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787\": rpc error: code = NotFound desc = could not find container \"8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787\": container with ID starting with 8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787 not found: ID does not exist"
Oct 30 19:07:36.075999 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.075970    4511 scope.go:122] "RemoveContainer" containerID="fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a"
Oct 30 19:07:36.076317 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.076296    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a"} err="failed to get container status \"fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a\": rpc error: code = NotFound desc = could not find container \"fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a\": container with ID starting with fd3af38a095b5907407af72812bd531e05157c217758893ce4d40adefad6536a not found: ID does not exist"
Oct 30 19:07:36.076379 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.076318    4511 scope.go:122] "RemoveContainer" containerID="8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787"
Oct 30 19:07:36.076584 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.076566    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787"} err="failed to get container status \"8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787\": rpc error: code = NotFound desc = could not find container \"8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787\": container with ID starting with 8640cef4bf65665ef144661ea842a70d4fff48113dfa6a06428da24987967787 not found: ID does not exist"
Oct 30 19:07:36.217478 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.217452    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"]
Oct 30 19:07:36.223229 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:36.223206    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/message-dumper-predictor-6476b87f8-lxmtg"]
Oct 30 19:07:37.655613 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:37.655582    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a3d5d69-7fab-4807-adda-f5fdc0ff2476" path="/var/lib/kubelet/pods/9a3d5d69-7fab-4807-adda-f5fdc0ff2476/volumes"
Oct 30 19:07:38.835300 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:38.835260    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8" podUID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Oct 30 19:07:40.078959 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.078751    4511 generic.go:356] "Generic (PLEG): container finished" podID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" containerID="71faa11797a77bad1014f5cb0a923f15226f5654df5590a71ea72a2409fc59b5" exitCode=0
Oct 30 19:07:40.078959 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.078844    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8" event={"ID":"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12","Type":"ContainerDied","Data":"71faa11797a77bad1014f5cb0a923f15226f5654df5590a71ea72a2409fc59b5"}
Oct 30 19:07:40.157654 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:07:40.157613    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:07:40.175824 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.175805    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:07:40.320045 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.320023    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kube-api-access-l9hbt\" (UniqueName: \"kubernetes.io/projected/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kube-api-access-l9hbt\") pod \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") "
Oct 30 19:07:40.320211 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.320079    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-proxy-tls\") pod \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") "
Oct 30 19:07:40.320211 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.320148    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kserve-provision-location\") pod \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") "
Oct 30 19:07:40.320211 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.320178    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-cabundle-cert\") pod \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") "
Oct 30 19:07:40.320394 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.320209    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kube-rbac-proxy-sar-config\") pod \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\" (UID: \"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12\") "
Oct 30 19:07:40.320544 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.320468    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kserve-provision-location" pod "9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" (UID: "9fcb1ec8-4e7a-4744-8bfa-d663e0272b12"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:40.320633 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.320553    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-cabundle-cert" pod "9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" (UID: "9fcb1ec8-4e7a-4744-8bfa-d663e0272b12"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:40.320633 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.320591    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kube-rbac-proxy-sar-config" pod "9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" (UID: "9fcb1ec8-4e7a-4744-8bfa-d663e0272b12"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:40.322296 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.322276    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-proxy-tls" pod "9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" (UID: "9fcb1ec8-4e7a-4744-8bfa-d663e0272b12"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:40.322392 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.322307    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kube-api-access-l9hbt" pod "9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" (UID: "9fcb1ec8-4e7a-4744-8bfa-d663e0272b12"). InnerVolumeSpecName "kube-api-access-l9hbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:40.421614 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.421582    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9hbt\" (UniqueName: \"kubernetes.io/projected/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kube-api-access-l9hbt\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:40.421614 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.421609    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:40.421794 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.421623    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:40.421794 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.421637    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:40.421794 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:40.421650    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:41.089020 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:41.088985    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8" event={"ID":"9fcb1ec8-4e7a-4744-8bfa-d663e0272b12","Type":"ContainerDied","Data":"dc4d5e6d9da3e62db4fd2230ab8d25e014456b576c234a21a5af1e4b190a9a74"}
Oct 30 19:07:41.089020 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:41.089020    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"
Oct 30 19:07:41.089561 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:41.089073    4511 scope.go:122] "RemoveContainer" containerID="d596f3a21ea843e687d5b7d5784d5cc77a0bece3a4a501ae7e95c93d201c0d62"
Oct 30 19:07:41.097610 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:41.097594    4511 scope.go:122] "RemoveContainer" containerID="71faa11797a77bad1014f5cb0a923f15226f5654df5590a71ea72a2409fc59b5"
Oct 30 19:07:41.104619 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:41.104602    4511 scope.go:122] "RemoveContainer" containerID="2a3eba5b9f07beb816307f524ddf4f457c2ef79e65570cd43de94fbfa01e7284"
Oct 30 19:07:41.293673 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:41.293643    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"]
Oct 30 19:07:41.340817 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:41.340750    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/sklearn-v2-mlserver-predictor-74989457cb-6t7f8"]
Oct 30 19:07:41.656317 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:07:41.656243    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" path="/var/lib/kubelet/pods/9fcb1ec8-4e7a-4744-8bfa-d663e0272b12/volumes"
Oct 30 19:08:08.631580 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.631539    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"]
Oct 30 19:08:08.632231 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.631949    4511 cpu_manager.go:414] "containerMap: removing container" podUID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" containerName="storage-initializer"
Oct 30 19:08:08.632231 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.631965    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" containerName="storage-initializer"
Oct 30 19:08:08.632231 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.631981    4511 cpu_manager.go:414] "containerMap: removing container" podUID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" containerName="kube-rbac-proxy"
Oct 30 19:08:08.632231 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.631990    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" containerName="kube-rbac-proxy"
Oct 30 19:08:08.632231 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.632018    4511 cpu_manager.go:414] "containerMap: removing container" podUID="9a3d5d69-7fab-4807-adda-f5fdc0ff2476" containerName="kube-rbac-proxy"
Oct 30 19:08:08.632231 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.632026    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a3d5d69-7fab-4807-adda-f5fdc0ff2476" containerName="kube-rbac-proxy"
Oct 30 19:08:08.632231 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.632038    4511 cpu_manager.go:414] "containerMap: removing container" podUID="9a3d5d69-7fab-4807-adda-f5fdc0ff2476" containerName="kserve-container"
Oct 30 19:08:08.632231 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.632044    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a3d5d69-7fab-4807-adda-f5fdc0ff2476" containerName="kserve-container"
Oct 30 19:08:08.632231 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.632060    4511 cpu_manager.go:414] "containerMap: removing container" podUID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" containerName="kserve-container"
Oct 30 19:08:08.632231 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.632067    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" containerName="kserve-container"
Oct 30 19:08:08.632231 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.632171    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a3d5d69-7fab-4807-adda-f5fdc0ff2476" containerName="kube-rbac-proxy"
Oct 30 19:08:08.632231 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.632185    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a3d5d69-7fab-4807-adda-f5fdc0ff2476" containerName="kserve-container"
Oct 30 19:08:08.632231 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.632195    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" containerName="kserve-container"
Oct 30 19:08:08.632231 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.632205    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fcb1ec8-4e7a-4744-8bfa-d663e0272b12" containerName="kube-rbac-proxy"
Oct 30 19:08:08.634375 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.634353    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:08:08.646160 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.646120    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779178cd-1a05-45e7-8d5a-c1aff47510d9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-5cf4588c5-qtx5z\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:08:08.646252 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.646208    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779178cd-1a05-45e7-8d5a-c1aff47510d9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-5cf4588c5-qtx5z\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:08:08.646252 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.646246    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rljm\" (UniqueName: \"kubernetes.io/projected/779178cd-1a05-45e7-8d5a-c1aff47510d9-kube-api-access-4rljm\") pod \"isvc-autogluon-v1-predictor-5cf4588c5-qtx5z\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:08:08.646320 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.646310    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/779178cd-1a05-45e7-8d5a-c1aff47510d9-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-5cf4588c5-qtx5z\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:08:08.646357 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.646333    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/779178cd-1a05-45e7-8d5a-c1aff47510d9-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-5cf4588c5-qtx5z\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:08:08.685579 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.685549    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:08:08.733057 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.733007    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:08:08.733057 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.733013    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Oct 30 19:08:08.733057 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.733041    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:08:08.733057 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.733023    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:08:08.733357 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.733015    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:08:08.733357 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.733229    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Oct 30 19:08:08.739451 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.739423    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"]
Oct 30 19:08:08.747383 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.747353    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779178cd-1a05-45e7-8d5a-c1aff47510d9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-5cf4588c5-qtx5z\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:08:08.747544 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.747393    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779178cd-1a05-45e7-8d5a-c1aff47510d9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-5cf4588c5-qtx5z\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:08:08.747544 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.747513    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rljm\" (UniqueName: \"kubernetes.io/projected/779178cd-1a05-45e7-8d5a-c1aff47510d9-kube-api-access-4rljm\") pod \"isvc-autogluon-v1-predictor-5cf4588c5-qtx5z\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:08:08.747660 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.747565    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/779178cd-1a05-45e7-8d5a-c1aff47510d9-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-5cf4588c5-qtx5z\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:08:08.747660 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.747591    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/779178cd-1a05-45e7-8d5a-c1aff47510d9-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-5cf4588c5-qtx5z\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:08:08.748049 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.748001    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/779178cd-1a05-45e7-8d5a-c1aff47510d9-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-5cf4588c5-qtx5z\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:08:08.748215 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.748189    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779178cd-1a05-45e7-8d5a-c1aff47510d9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-5cf4588c5-qtx5z\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:08:08.748297 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.748245    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/779178cd-1a05-45e7-8d5a-c1aff47510d9-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-5cf4588c5-qtx5z\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:08:08.750106 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.750086    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779178cd-1a05-45e7-8d5a-c1aff47510d9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-5cf4588c5-qtx5z\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:08:08.808760 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.808726    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rljm\" (UniqueName: \"kubernetes.io/projected/779178cd-1a05-45e7-8d5a-c1aff47510d9-kube-api-access-4rljm\") pod \"isvc-autogluon-v1-predictor-5cf4588c5-qtx5z\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:08:08.948368 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:08.948272    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:08:09.374640 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:09.374605    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"]
Oct 30 19:08:09.381629 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:08:09.381599    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779178cd_1a05_45e7_8d5a_c1aff47510d9.slice/crio-d6de8d3f07285805a1e99a142a4e00fd9e95b1c0b5a4fbc9f7c96c1cb71445d2 WatchSource:0}: Error finding container d6de8d3f07285805a1e99a142a4e00fd9e95b1c0b5a4fbc9f7c96c1cb71445d2: Status 404 returned error can't find the container with id d6de8d3f07285805a1e99a142a4e00fd9e95b1c0b5a4fbc9f7c96c1cb71445d2
Oct 30 19:08:10.158039 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:08:10.158003    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:08:10.200153 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:10.200090    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z" event={"ID":"779178cd-1a05-45e7-8d5a-c1aff47510d9","Type":"ContainerStarted","Data":"f418329c93f075cd055839ea3ff326153993c584b62822683c79f89fca5b966e"}
Oct 30 19:08:10.200326 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:10.200166    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z" event={"ID":"779178cd-1a05-45e7-8d5a-c1aff47510d9","Type":"ContainerStarted","Data":"d6de8d3f07285805a1e99a142a4e00fd9e95b1c0b5a4fbc9f7c96c1cb71445d2"}
Oct 30 19:08:22.722695 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.722655    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"]
Oct 30 19:08:22.725381 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.725355    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:22.744700 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.744668    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:22.744825 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.744797    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phnxb\" (UniqueName: \"kubernetes.io/projected/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kube-api-access-phnxb\") pod \"isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:22.744864 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.744851    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:22.744919 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.744902    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:22.745162 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.745119    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:22.775652 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.775610    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:08:22.775652 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.775616    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:08:22.775652 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.775641    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:08:22.822772 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.822741    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:08:22.822933 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.822782    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:08:22.822933 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.822799    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Oct 30 19:08:22.822933 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.822780    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:08:22.828788 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.828766    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"]
Oct 30 19:08:22.845878 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.845842    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:22.846023 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.845899    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:22.846023 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.845988    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phnxb\" (UniqueName: \"kubernetes.io/projected/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kube-api-access-phnxb\") pod \"isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:22.846158 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.846035    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:22.846158 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.846075    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:22.846567 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.846535    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:22.846701 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.846662    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:22.846825 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.846804    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:22.848796 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.848767    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:22.904882 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:22.904852    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phnxb\" (UniqueName: \"kubernetes.io/projected/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kube-api-access-phnxb\") pod \"isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:23.039726 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:23.039634    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:23.460608 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:23.460571    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"]
Oct 30 19:08:23.468527 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:08:23.468501    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e6d5c0_2701_4ceb_b92a_7e314a70cc3d.slice/crio-5ad64a42971b976dbcc825ea564479a10a033d21db6352dfb8681de6fb5bf610 WatchSource:0}: Error finding container 5ad64a42971b976dbcc825ea564479a10a033d21db6352dfb8681de6fb5bf610: Status 404 returned error can't find the container with id 5ad64a42971b976dbcc825ea564479a10a033d21db6352dfb8681de6fb5bf610
Oct 30 19:08:24.259769 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:24.259725    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b" event={"ID":"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d","Type":"ContainerStarted","Data":"0096201da391bd51e66d997f9df5ccbee2413505e8ceee4e02145d9654b45791"}
Oct 30 19:08:24.260236 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:24.259791    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b" event={"ID":"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d","Type":"ContainerStarted","Data":"5ad64a42971b976dbcc825ea564479a10a033d21db6352dfb8681de6fb5bf610"}
Oct 30 19:08:25.767988 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:25.767854    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 19:08:25.775515 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:25.773680    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 19:08:27.272664 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:27.272635    4511 generic.go:356] "Generic (PLEG): container finished" podID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" containerID="0096201da391bd51e66d997f9df5ccbee2413505e8ceee4e02145d9654b45791" exitCode=0
Oct 30 19:08:27.273010 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:27.272699    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b" event={"ID":"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d","Type":"ContainerDied","Data":"0096201da391bd51e66d997f9df5ccbee2413505e8ceee4e02145d9654b45791"}
Oct 30 19:08:28.278763 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:28.278730    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b" event={"ID":"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d","Type":"ContainerStarted","Data":"b1545abd92da0f148de7ca70e6ef4aa2d50f537141e70cfd9cb5b7511b1ba3c1"}
Oct 30 19:08:28.279195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:28.278791    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b" event={"ID":"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d","Type":"ContainerStarted","Data":"81d8a95fbccf16989205e8d393350d096daf2a6ea9062997b4418b9befce28f0"}
Oct 30 19:08:28.279195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:28.278981    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:28.394822 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:28.394770    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b" podStartSLOduration=2.658885165 podStartE2EDuration="6.394756592s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.735871427s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:28.393893773 +0000 UTC m=+1802.911578674" watchObservedRunningTime="2026-10-30 19:08:28.394756592 +0000 UTC m=+1802.912441492"
Oct 30 19:08:29.282851 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:29.282812    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:35.292409 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:35.292380    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:08:40.158773 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:08:40.158730    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:08:53.377820 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:53.377784    4511 generic.go:356] "Generic (PLEG): container finished" podID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerID="f418329c93f075cd055839ea3ff326153993c584b62822683c79f89fca5b966e" exitCode=0
Oct 30 19:08:53.378217 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:08:53.377853    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z" event={"ID":"779178cd-1a05-45e7-8d5a-c1aff47510d9","Type":"ContainerDied","Data":"f418329c93f075cd055839ea3ff326153993c584b62822683c79f89fca5b966e"}
Oct 30 19:09:05.318100 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:05.318044    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b" podUID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 30 19:09:10.160978 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:09:10.160888    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:09:15.296032 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:15.295998    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:09:22.867698 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:22.867660    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"]
Oct 30 19:09:22.868233 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:22.867984    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b" podUID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" containerName="kserve-container" containerID="cri-o://81d8a95fbccf16989205e8d393350d096daf2a6ea9062997b4418b9befce28f0" gracePeriod=30
Oct 30 19:09:22.868233 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:22.867993    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b" podUID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" containerName="kube-rbac-proxy" containerID="cri-o://b1545abd92da0f148de7ca70e6ef4aa2d50f537141e70cfd9cb5b7511b1ba3c1" gracePeriod=30
Oct 30 19:09:23.524687 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:23.524649    4511 generic.go:356] "Generic (PLEG): container finished" podID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" containerID="b1545abd92da0f148de7ca70e6ef4aa2d50f537141e70cfd9cb5b7511b1ba3c1" exitCode=0
Oct 30 19:09:23.524876 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:23.524716    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b" event={"ID":"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d","Type":"ContainerDied","Data":"b1545abd92da0f148de7ca70e6ef4aa2d50f537141e70cfd9cb5b7511b1ba3c1"}
Oct 30 19:09:25.287701 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:25.287662    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b" podUID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Oct 30 19:09:26.334317 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:26.334273    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b" podUID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.38:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:30.287763 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:30.287716    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b" podUID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Oct 30 19:09:31.039778 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.039744    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:09:31.109306 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.109167    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kserve-provision-location\") pod \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") "
Oct 30 19:09:31.109306 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.109226    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-proxy-tls\") pod \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") "
Oct 30 19:09:31.109306 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.109265    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kube-api-access-phnxb\" (UniqueName: \"kubernetes.io/projected/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kube-api-access-phnxb\") pod \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") "
Oct 30 19:09:31.109306 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.109288    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-cabundle-cert\") pod \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") "
Oct 30 19:09:31.109672 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.109314    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kube-rbac-proxy-sar-config\") pod \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\" (UID: \"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d\") "
Oct 30 19:09:31.109964 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.109757    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-cabundle-cert" pod "89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" (UID: "89e6d5c0-2701-4ceb-b92a-7e314a70cc3d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:31.109964 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.109760    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kube-rbac-proxy-sar-config" pod "89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" (UID: "89e6d5c0-2701-4ceb-b92a-7e314a70cc3d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:31.109964 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.109862    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kserve-provision-location" pod "89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" (UID: "89e6d5c0-2701-4ceb-b92a-7e314a70cc3d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:31.112005 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.111972    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kube-api-access-phnxb" pod "89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" (UID: "89e6d5c0-2701-4ceb-b92a-7e314a70cc3d"). InnerVolumeSpecName "kube-api-access-phnxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:31.112639 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.112606    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-proxy-tls" pod "89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" (UID: "89e6d5c0-2701-4ceb-b92a-7e314a70cc3d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:09:31.210552 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.210513    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:31.210552 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.210550    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:31.210552 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.210561    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phnxb\" (UniqueName: \"kubernetes.io/projected/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kube-api-access-phnxb\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:31.210847 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.210570    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:31.210847 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.210579    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:31.562432 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.562397    4511 generic.go:356] "Generic (PLEG): container finished" podID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" containerID="81d8a95fbccf16989205e8d393350d096daf2a6ea9062997b4418b9befce28f0" exitCode=0
Oct 30 19:09:31.562887 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.562466    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"
Oct 30 19:09:31.562887 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.562486    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b" event={"ID":"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d","Type":"ContainerDied","Data":"81d8a95fbccf16989205e8d393350d096daf2a6ea9062997b4418b9befce28f0"}
Oct 30 19:09:31.562887 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.562559    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b" event={"ID":"89e6d5c0-2701-4ceb-b92a-7e314a70cc3d","Type":"ContainerDied","Data":"5ad64a42971b976dbcc825ea564479a10a033d21db6352dfb8681de6fb5bf610"}
Oct 30 19:09:31.562887 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.562609    4511 scope.go:122] "RemoveContainer" containerID="b1545abd92da0f148de7ca70e6ef4aa2d50f537141e70cfd9cb5b7511b1ba3c1"
Oct 30 19:09:31.572962 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.572946    4511 scope.go:122] "RemoveContainer" containerID="81d8a95fbccf16989205e8d393350d096daf2a6ea9062997b4418b9befce28f0"
Oct 30 19:09:31.581672 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.581654    4511 scope.go:122] "RemoveContainer" containerID="0096201da391bd51e66d997f9df5ccbee2413505e8ceee4e02145d9654b45791"
Oct 30 19:09:31.590072 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.590040    4511 scope.go:122] "RemoveContainer" containerID="b1545abd92da0f148de7ca70e6ef4aa2d50f537141e70cfd9cb5b7511b1ba3c1"
Oct 30 19:09:31.590387 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:09:31.590362    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1545abd92da0f148de7ca70e6ef4aa2d50f537141e70cfd9cb5b7511b1ba3c1\": container with ID starting with b1545abd92da0f148de7ca70e6ef4aa2d50f537141e70cfd9cb5b7511b1ba3c1 not found: ID does not exist" containerID="b1545abd92da0f148de7ca70e6ef4aa2d50f537141e70cfd9cb5b7511b1ba3c1"
Oct 30 19:09:31.590472 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.590392    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1545abd92da0f148de7ca70e6ef4aa2d50f537141e70cfd9cb5b7511b1ba3c1"} err="failed to get container status \"b1545abd92da0f148de7ca70e6ef4aa2d50f537141e70cfd9cb5b7511b1ba3c1\": rpc error: code = NotFound desc = could not find container \"b1545abd92da0f148de7ca70e6ef4aa2d50f537141e70cfd9cb5b7511b1ba3c1\": container with ID starting with b1545abd92da0f148de7ca70e6ef4aa2d50f537141e70cfd9cb5b7511b1ba3c1 not found: ID does not exist"
Oct 30 19:09:31.590472 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.590413    4511 scope.go:122] "RemoveContainer" containerID="81d8a95fbccf16989205e8d393350d096daf2a6ea9062997b4418b9befce28f0"
Oct 30 19:09:31.590679 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:09:31.590651    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d8a95fbccf16989205e8d393350d096daf2a6ea9062997b4418b9befce28f0\": container with ID starting with 81d8a95fbccf16989205e8d393350d096daf2a6ea9062997b4418b9befce28f0 not found: ID does not exist" containerID="81d8a95fbccf16989205e8d393350d096daf2a6ea9062997b4418b9befce28f0"
Oct 30 19:09:31.590790 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.590687    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d8a95fbccf16989205e8d393350d096daf2a6ea9062997b4418b9befce28f0"} err="failed to get container status \"81d8a95fbccf16989205e8d393350d096daf2a6ea9062997b4418b9befce28f0\": rpc error: code = NotFound desc = could not find container \"81d8a95fbccf16989205e8d393350d096daf2a6ea9062997b4418b9befce28f0\": container with ID starting with 81d8a95fbccf16989205e8d393350d096daf2a6ea9062997b4418b9befce28f0 not found: ID does not exist"
Oct 30 19:09:31.590790 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.590709    4511 scope.go:122] "RemoveContainer" containerID="0096201da391bd51e66d997f9df5ccbee2413505e8ceee4e02145d9654b45791"
Oct 30 19:09:31.590994 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:09:31.590972    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0096201da391bd51e66d997f9df5ccbee2413505e8ceee4e02145d9654b45791\": container with ID starting with 0096201da391bd51e66d997f9df5ccbee2413505e8ceee4e02145d9654b45791 not found: ID does not exist" containerID="0096201da391bd51e66d997f9df5ccbee2413505e8ceee4e02145d9654b45791"
Oct 30 19:09:31.591033 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.591002    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0096201da391bd51e66d997f9df5ccbee2413505e8ceee4e02145d9654b45791"} err="failed to get container status \"0096201da391bd51e66d997f9df5ccbee2413505e8ceee4e02145d9654b45791\": rpc error: code = NotFound desc = could not find container \"0096201da391bd51e66d997f9df5ccbee2413505e8ceee4e02145d9654b45791\": container with ID starting with 0096201da391bd51e66d997f9df5ccbee2413505e8ceee4e02145d9654b45791 not found: ID does not exist"
Oct 30 19:09:31.720090 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.720055    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"]
Oct 30 19:09:31.767400 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:31.767367    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-v2-runtime-predictor-9bdc4f56d-7mm5b"]
Oct 30 19:09:33.658098 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:09:33.658062    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" path="/var/lib/kubelet/pods/89e6d5c0-2701-4ceb-b92a-7e314a70cc3d/volumes"
Oct 30 19:09:40.161342 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:09:40.161305    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:10:10.211263 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:10:10.211218    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:10:40.156807 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:10:40.156723    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:10:40.861685 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:10:40.861651    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z" event={"ID":"779178cd-1a05-45e7-8d5a-c1aff47510d9","Type":"ContainerStarted","Data":"e9cb49f821c32f874a801b4e4104e2d5a6ef768b806f4f0549037066baf80d75"}
Oct 30 19:10:40.861883 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:10:40.861732    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z" event={"ID":"779178cd-1a05-45e7-8d5a-c1aff47510d9","Type":"ContainerStarted","Data":"e7c58bee986321fca8c98013497d1aef044e973a968ece45fa47647a7734646b"}
Oct 30 19:10:40.861883 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:10:40.861764    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:10:40.861883 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:10:40.861818    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:10:40.863466 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:10:40.863443    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z" podUID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Oct 30 19:10:41.866633 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:10:41.866589    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z" podUID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Oct 30 19:10:46.871478 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:10:46.871446    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:10:46.872021 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:10:46.871991    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z" podUID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Oct 30 19:10:47.034622 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:10:47.034567    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z" podStartSLOduration=8.652002601 podStartE2EDuration="2m39.034554123s" totalImagesPullingTime="1m47.073593234s" totalInitContainerRuntime="43.308958288s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:41.02443832 +0000 UTC m=+1935.542123220" watchObservedRunningTime="2026-10-30 19:10:47.034554123 +0000 UTC m=+1941.552239020"
Oct 30 19:10:56.873147 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:10:56.873106    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:11:10.157200 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:11:10.157145    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:11:15.169099 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:15.169062    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"]
Oct 30 19:11:15.169661 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:15.169313    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z" podUID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerName="kserve-container" containerID="cri-o://e7c58bee986321fca8c98013497d1aef044e973a968ece45fa47647a7734646b" gracePeriod=30
Oct 30 19:11:15.169661 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:15.169330    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z" podUID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerName="kube-rbac-proxy" containerID="cri-o://e9cb49f821c32f874a801b4e4104e2d5a6ef768b806f4f0549037066baf80d75" gracePeriod=30
Oct 30 19:11:15.993643 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:15.993605    4511 generic.go:356] "Generic (PLEG): container finished" podID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerID="e9cb49f821c32f874a801b4e4104e2d5a6ef768b806f4f0549037066baf80d75" exitCode=0
Oct 30 19:11:15.993826 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:15.993685    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z" event={"ID":"779178cd-1a05-45e7-8d5a-c1aff47510d9","Type":"ContainerDied","Data":"e9cb49f821c32f874a801b4e4104e2d5a6ef768b806f4f0549037066baf80d75"}
Oct 30 19:11:16.867897 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:16.867850    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z" podUID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Oct 30 19:11:16.872167 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:16.872111    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z" podUID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Oct 30 19:11:20.010825 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.010733    4511 generic.go:356] "Generic (PLEG): container finished" podID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerID="e7c58bee986321fca8c98013497d1aef044e973a968ece45fa47647a7734646b" exitCode=0
Oct 30 19:11:20.010825 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.010802    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z" event={"ID":"779178cd-1a05-45e7-8d5a-c1aff47510d9","Type":"ContainerDied","Data":"e7c58bee986321fca8c98013497d1aef044e973a968ece45fa47647a7734646b"}
Oct 30 19:11:20.041282 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.041253    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:11:20.221303 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.221266    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/779178cd-1a05-45e7-8d5a-c1aff47510d9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779178cd-1a05-45e7-8d5a-c1aff47510d9-kube-rbac-proxy-sar-config\") pod \"779178cd-1a05-45e7-8d5a-c1aff47510d9\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") "
Oct 30 19:11:20.221476 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.221322    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/779178cd-1a05-45e7-8d5a-c1aff47510d9-kube-api-access-4rljm\" (UniqueName: \"kubernetes.io/projected/779178cd-1a05-45e7-8d5a-c1aff47510d9-kube-api-access-4rljm\") pod \"779178cd-1a05-45e7-8d5a-c1aff47510d9\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") "
Oct 30 19:11:20.221476 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.221352    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/779178cd-1a05-45e7-8d5a-c1aff47510d9-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/779178cd-1a05-45e7-8d5a-c1aff47510d9-cabundle-cert\") pod \"779178cd-1a05-45e7-8d5a-c1aff47510d9\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") "
Oct 30 19:11:20.221476 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.221396    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/779178cd-1a05-45e7-8d5a-c1aff47510d9-proxy-tls\" (UniqueName: \"kubernetes.io/secret/779178cd-1a05-45e7-8d5a-c1aff47510d9-proxy-tls\") pod \"779178cd-1a05-45e7-8d5a-c1aff47510d9\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") "
Oct 30 19:11:20.221476 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.221439    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779178cd-1a05-45e7-8d5a-c1aff47510d9-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/779178cd-1a05-45e7-8d5a-c1aff47510d9-kserve-provision-location\") pod \"779178cd-1a05-45e7-8d5a-c1aff47510d9\" (UID: \"779178cd-1a05-45e7-8d5a-c1aff47510d9\") "
Oct 30 19:11:20.221711 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.221686    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779178cd-1a05-45e7-8d5a-c1aff47510d9-kube-rbac-proxy-sar-config" pod "779178cd-1a05-45e7-8d5a-c1aff47510d9" (UID: "779178cd-1a05-45e7-8d5a-c1aff47510d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:20.221813 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.221794    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779178cd-1a05-45e7-8d5a-c1aff47510d9-cabundle-cert" pod "779178cd-1a05-45e7-8d5a-c1aff47510d9" (UID: "779178cd-1a05-45e7-8d5a-c1aff47510d9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:20.223670 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.223649    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779178cd-1a05-45e7-8d5a-c1aff47510d9-proxy-tls" pod "779178cd-1a05-45e7-8d5a-c1aff47510d9" (UID: "779178cd-1a05-45e7-8d5a-c1aff47510d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:20.223941 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.223924    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779178cd-1a05-45e7-8d5a-c1aff47510d9-kube-api-access-4rljm" pod "779178cd-1a05-45e7-8d5a-c1aff47510d9" (UID: "779178cd-1a05-45e7-8d5a-c1aff47510d9"). InnerVolumeSpecName "kube-api-access-4rljm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:20.225521 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.225495    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779178cd-1a05-45e7-8d5a-c1aff47510d9-kserve-provision-location" pod "779178cd-1a05-45e7-8d5a-c1aff47510d9" (UID: "779178cd-1a05-45e7-8d5a-c1aff47510d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:11:20.322383 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.322343    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779178cd-1a05-45e7-8d5a-c1aff47510d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:20.322383 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.322382    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rljm\" (UniqueName: \"kubernetes.io/projected/779178cd-1a05-45e7-8d5a-c1aff47510d9-kube-api-access-4rljm\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:20.322383 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.322394    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/779178cd-1a05-45e7-8d5a-c1aff47510d9-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:20.322628 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.322402    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779178cd-1a05-45e7-8d5a-c1aff47510d9-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:20.322628 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:20.322412    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/779178cd-1a05-45e7-8d5a-c1aff47510d9-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:21.022492 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:21.022455    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z" event={"ID":"779178cd-1a05-45e7-8d5a-c1aff47510d9","Type":"ContainerDied","Data":"d6de8d3f07285805a1e99a142a4e00fd9e95b1c0b5a4fbc9f7c96c1cb71445d2"}
Oct 30 19:11:21.022946 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:21.022548    4511 scope.go:122] "RemoveContainer" containerID="e9cb49f821c32f874a801b4e4104e2d5a6ef768b806f4f0549037066baf80d75"
Oct 30 19:11:21.022946 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:21.022465    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"
Oct 30 19:11:21.031006 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:21.030982    4511 scope.go:122] "RemoveContainer" containerID="e7c58bee986321fca8c98013497d1aef044e973a968ece45fa47647a7734646b"
Oct 30 19:11:21.038496 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:21.038482    4511 scope.go:122] "RemoveContainer" containerID="f418329c93f075cd055839ea3ff326153993c584b62822683c79f89fca5b966e"
Oct 30 19:11:21.227748 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:21.227709    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"]
Oct 30 19:11:21.275081 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:21.275008    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v1-predictor-5cf4588c5-qtx5z"]
Oct 30 19:11:21.655769 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:21.655728    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="779178cd-1a05-45e7-8d5a-c1aff47510d9" path="/var/lib/kubelet/pods/779178cd-1a05-45e7-8d5a-c1aff47510d9/volumes"
Oct 30 19:11:23.451443 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.451412    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"]
Oct 30 19:11:23.451838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.451685    4511 cpu_manager.go:414] "containerMap: removing container" podUID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerName="storage-initializer"
Oct 30 19:11:23.451838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.451695    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerName="storage-initializer"
Oct 30 19:11:23.451838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.451704    4511 cpu_manager.go:414] "containerMap: removing container" podUID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerName="kube-rbac-proxy"
Oct 30 19:11:23.451838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.451711    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerName="kube-rbac-proxy"
Oct 30 19:11:23.451838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.451723    4511 cpu_manager.go:414] "containerMap: removing container" podUID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" containerName="kserve-container"
Oct 30 19:11:23.451838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.451727    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" containerName="kserve-container"
Oct 30 19:11:23.451838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.451736    4511 cpu_manager.go:414] "containerMap: removing container" podUID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerName="kserve-container"
Oct 30 19:11:23.451838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.451740    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerName="kserve-container"
Oct 30 19:11:23.451838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.451748    4511 cpu_manager.go:414] "containerMap: removing container" podUID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" containerName="storage-initializer"
Oct 30 19:11:23.451838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.451752    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" containerName="storage-initializer"
Oct 30 19:11:23.451838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.451760    4511 cpu_manager.go:414] "containerMap: removing container" podUID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" containerName="kube-rbac-proxy"
Oct 30 19:11:23.451838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.451765    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" containerName="kube-rbac-proxy"
Oct 30 19:11:23.451838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.451813    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" containerName="kube-rbac-proxy"
Oct 30 19:11:23.451838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.451822    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerName="kube-rbac-proxy"
Oct 30 19:11:23.451838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.451828    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e6d5c0-2701-4ceb-b92a-7e314a70cc3d" containerName="kserve-container"
Oct 30 19:11:23.451838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.451834    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="779178cd-1a05-45e7-8d5a-c1aff47510d9" containerName="kserve-container"
Oct 30 19:11:23.453895 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.453877    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:23.504483 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.504439    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Oct 30 19:11:23.551603 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.551563    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Oct 30 19:11:23.551793 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.551729    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:11:23.551793 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.551740    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:11:23.551793 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.551729    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:11:23.551793 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.551756    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:11:23.551793 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.551780    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:11:23.558804 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.558776    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"]
Oct 30 19:11:23.651661 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.651630    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc7kq\" (UniqueName: \"kubernetes.io/projected/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kube-api-access-vc7kq\") pod \"isvc-autogluon-v2-predictor-75cc48778f-dfcmb\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:23.651855 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.651738    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-proxy-tls\") pod \"isvc-autogluon-v2-predictor-75cc48778f-dfcmb\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:23.651855 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.651805    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-75cc48778f-dfcmb\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:23.651972 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.651863    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-75cc48778f-dfcmb\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:23.651972 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.651906    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-75cc48778f-dfcmb\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:23.753306 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.753227    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-75cc48778f-dfcmb\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:23.753306 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.753270    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-75cc48778f-dfcmb\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:23.753501 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.753311    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc7kq\" (UniqueName: \"kubernetes.io/projected/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kube-api-access-vc7kq\") pod \"isvc-autogluon-v2-predictor-75cc48778f-dfcmb\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:23.753501 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.753339    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-proxy-tls\") pod \"isvc-autogluon-v2-predictor-75cc48778f-dfcmb\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:23.753501 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.753361    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-75cc48778f-dfcmb\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:23.753812 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.753780    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-75cc48778f-dfcmb\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:23.754032 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.754014    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-75cc48778f-dfcmb\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:23.754112 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.754045    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-75cc48778f-dfcmb\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:23.755947 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.755931    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-proxy-tls\") pod \"isvc-autogluon-v2-predictor-75cc48778f-dfcmb\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:23.812505 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:23.812473    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc7kq\" (UniqueName: \"kubernetes.io/projected/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kube-api-access-vc7kq\") pod \"isvc-autogluon-v2-predictor-75cc48778f-dfcmb\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:24.066888 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:24.066859    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:24.487864 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:24.487780    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"]
Oct 30 19:11:24.495727 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:11:24.495693    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8094f5_a0d0_41ae_bc8b_3d59ede67e87.slice/crio-41ff69ec8f0c2fe50b55378e1871ae1c2cbacbf66fd592226a6ecfddf428095c WatchSource:0}: Error finding container 41ff69ec8f0c2fe50b55378e1871ae1c2cbacbf66fd592226a6ecfddf428095c: Status 404 returned error can't find the container with id 41ff69ec8f0c2fe50b55378e1871ae1c2cbacbf66fd592226a6ecfddf428095c
Oct 30 19:11:25.041900 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:25.041860    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb" event={"ID":"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87","Type":"ContainerStarted","Data":"219363b920f82253891db51c4292ff9bb82d915590b830616aadef32af46fbd7"}
Oct 30 19:11:25.042073 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:25.041916    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb" event={"ID":"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87","Type":"ContainerStarted","Data":"41ff69ec8f0c2fe50b55378e1871ae1c2cbacbf66fd592226a6ecfddf428095c"}
Oct 30 19:11:31.064385 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:31.064356    4511 generic.go:356] "Generic (PLEG): container finished" podID="1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" containerID="219363b920f82253891db51c4292ff9bb82d915590b830616aadef32af46fbd7" exitCode=0
Oct 30 19:11:31.064714 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:31.064434    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb" event={"ID":"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87","Type":"ContainerDied","Data":"219363b920f82253891db51c4292ff9bb82d915590b830616aadef32af46fbd7"}
Oct 30 19:11:32.070767 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:32.070733    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb" event={"ID":"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87","Type":"ContainerStarted","Data":"da59b686dcb76b9068d99024a015778cbfd1edb5cc43248f3ad22a379785619a"}
Oct 30 19:11:32.071194 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:32.070791    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb" event={"ID":"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87","Type":"ContainerStarted","Data":"aa7e5eb837592ba39baf8e6ee2dc409fb8e833ba62325b160f591d653d0b6a38"}
Oct 30 19:11:32.076677 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:32.071448    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:32.076677 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:32.071552    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:32.185968 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:32.185918    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb" podStartSLOduration=2.79417923 podStartE2EDuration="9.185904384s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.391725154s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:32.185056339 +0000 UTC m=+1986.702741249" watchObservedRunningTime="2026-10-30 19:11:32.185904384 +0000 UTC m=+1986.703589284"
Oct 30 19:11:38.081273 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:11:38.081241    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:11:40.157806 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:11:40.157765    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:12:10.156140 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:12:10.156046    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:12:40.156101 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:12:40.156065    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:13:05.685786 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:05.685751    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:13:10.156223 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:13:10.156184    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:13:16.694329 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:16.694288    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb" podUID="1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.39:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:13:19.884946 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:19.884915    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"]
Oct 30 19:13:19.885368 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:19.885172    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb" podUID="1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" containerName="kserve-container" containerID="cri-o://aa7e5eb837592ba39baf8e6ee2dc409fb8e833ba62325b160f591d653d0b6a38" gracePeriod=30
Oct 30 19:13:19.885368 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:19.885209    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb" podUID="1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" containerName="kube-rbac-proxy" containerID="cri-o://da59b686dcb76b9068d99024a015778cbfd1edb5cc43248f3ad22a379785619a" gracePeriod=30
Oct 30 19:13:20.458145 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:20.458096    4511 generic.go:356] "Generic (PLEG): container finished" podID="1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" containerID="da59b686dcb76b9068d99024a015778cbfd1edb5cc43248f3ad22a379785619a" exitCode=0
Oct 30 19:13:20.458332 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:20.458173    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb" event={"ID":"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87","Type":"ContainerDied","Data":"da59b686dcb76b9068d99024a015778cbfd1edb5cc43248f3ad22a379785619a"}
Oct 30 19:13:21.324355 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.324326    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:13:21.433226 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.433164    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-proxy-tls\") pod \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") "
Oct 30 19:13:21.433226 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.433198    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-cabundle-cert\") pod \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") "
Oct 30 19:13:21.433402 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.433240    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kube-api-access-vc7kq\" (UniqueName: \"kubernetes.io/projected/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kube-api-access-vc7kq\") pod \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") "
Oct 30 19:13:21.433402 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.433268    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kserve-provision-location\") pod \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") "
Oct 30 19:13:21.433402 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.433298    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kube-rbac-proxy-sar-config\") pod \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\" (UID: \"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87\") "
Oct 30 19:13:21.433607 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.433580    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-cabundle-cert" pod "1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" (UID: "1f8094f5-a0d0-41ae-bc8b-3d59ede67e87"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:21.433719 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.433698    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kube-rbac-proxy-sar-config" pod "1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" (UID: "1f8094f5-a0d0-41ae-bc8b-3d59ede67e87"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:21.435512 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.435488    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kube-api-access-vc7kq" pod "1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" (UID: "1f8094f5-a0d0-41ae-bc8b-3d59ede67e87"). InnerVolumeSpecName "kube-api-access-vc7kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:21.435773 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.435757    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-proxy-tls" pod "1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" (UID: "1f8094f5-a0d0-41ae-bc8b-3d59ede67e87"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:21.438825 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.438799    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kserve-provision-location" pod "1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" (UID: "1f8094f5-a0d0-41ae-bc8b-3d59ede67e87"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.463633 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.463608    4511 generic.go:356] "Generic (PLEG): container finished" podID="1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" containerID="aa7e5eb837592ba39baf8e6ee2dc409fb8e833ba62325b160f591d653d0b6a38" exitCode=0
Oct 30 19:13:21.463749 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.463654    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb" event={"ID":"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87","Type":"ContainerDied","Data":"aa7e5eb837592ba39baf8e6ee2dc409fb8e833ba62325b160f591d653d0b6a38"}
Oct 30 19:13:21.463749 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.463680    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"
Oct 30 19:13:21.463749 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.463698    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb" event={"ID":"1f8094f5-a0d0-41ae-bc8b-3d59ede67e87","Type":"ContainerDied","Data":"41ff69ec8f0c2fe50b55378e1871ae1c2cbacbf66fd592226a6ecfddf428095c"}
Oct 30 19:13:21.463749 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.463725    4511 scope.go:122] "RemoveContainer" containerID="da59b686dcb76b9068d99024a015778cbfd1edb5cc43248f3ad22a379785619a"
Oct 30 19:13:21.472394 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.472378    4511 scope.go:122] "RemoveContainer" containerID="aa7e5eb837592ba39baf8e6ee2dc409fb8e833ba62325b160f591d653d0b6a38"
Oct 30 19:13:21.480328 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.480308    4511 scope.go:122] "RemoveContainer" containerID="219363b920f82253891db51c4292ff9bb82d915590b830616aadef32af46fbd7"
Oct 30 19:13:21.487678 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.487661    4511 scope.go:122] "RemoveContainer" containerID="da59b686dcb76b9068d99024a015778cbfd1edb5cc43248f3ad22a379785619a"
Oct 30 19:13:21.487993 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:13:21.487971    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da59b686dcb76b9068d99024a015778cbfd1edb5cc43248f3ad22a379785619a\": container with ID starting with da59b686dcb76b9068d99024a015778cbfd1edb5cc43248f3ad22a379785619a not found: ID does not exist" containerID="da59b686dcb76b9068d99024a015778cbfd1edb5cc43248f3ad22a379785619a"
Oct 30 19:13:21.488206 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.488006    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da59b686dcb76b9068d99024a015778cbfd1edb5cc43248f3ad22a379785619a"} err="failed to get container status \"da59b686dcb76b9068d99024a015778cbfd1edb5cc43248f3ad22a379785619a\": rpc error: code = NotFound desc = could not find container \"da59b686dcb76b9068d99024a015778cbfd1edb5cc43248f3ad22a379785619a\": container with ID starting with da59b686dcb76b9068d99024a015778cbfd1edb5cc43248f3ad22a379785619a not found: ID does not exist"
Oct 30 19:13:21.488353 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.488210    4511 scope.go:122] "RemoveContainer" containerID="aa7e5eb837592ba39baf8e6ee2dc409fb8e833ba62325b160f591d653d0b6a38"
Oct 30 19:13:21.488613 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:13:21.488591    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7e5eb837592ba39baf8e6ee2dc409fb8e833ba62325b160f591d653d0b6a38\": container with ID starting with aa7e5eb837592ba39baf8e6ee2dc409fb8e833ba62325b160f591d653d0b6a38 not found: ID does not exist" containerID="aa7e5eb837592ba39baf8e6ee2dc409fb8e833ba62325b160f591d653d0b6a38"
Oct 30 19:13:21.488683 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.488625    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7e5eb837592ba39baf8e6ee2dc409fb8e833ba62325b160f591d653d0b6a38"} err="failed to get container status \"aa7e5eb837592ba39baf8e6ee2dc409fb8e833ba62325b160f591d653d0b6a38\": rpc error: code = NotFound desc = could not find container \"aa7e5eb837592ba39baf8e6ee2dc409fb8e833ba62325b160f591d653d0b6a38\": container with ID starting with aa7e5eb837592ba39baf8e6ee2dc409fb8e833ba62325b160f591d653d0b6a38 not found: ID does not exist"
Oct 30 19:13:21.488683 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.488645    4511 scope.go:122] "RemoveContainer" containerID="219363b920f82253891db51c4292ff9bb82d915590b830616aadef32af46fbd7"
Oct 30 19:13:21.488927 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:13:21.488908    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219363b920f82253891db51c4292ff9bb82d915590b830616aadef32af46fbd7\": container with ID starting with 219363b920f82253891db51c4292ff9bb82d915590b830616aadef32af46fbd7 not found: ID does not exist" containerID="219363b920f82253891db51c4292ff9bb82d915590b830616aadef32af46fbd7"
Oct 30 19:13:21.488988 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.488935    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219363b920f82253891db51c4292ff9bb82d915590b830616aadef32af46fbd7"} err="failed to get container status \"219363b920f82253891db51c4292ff9bb82d915590b830616aadef32af46fbd7\": rpc error: code = NotFound desc = could not find container \"219363b920f82253891db51c4292ff9bb82d915590b830616aadef32af46fbd7\": container with ID starting with 219363b920f82253891db51c4292ff9bb82d915590b830616aadef32af46fbd7 not found: ID does not exist"
Oct 30 19:13:21.534684 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.534650    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc7kq\" (UniqueName: \"kubernetes.io/projected/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kube-api-access-vc7kq\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.534684 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.534677    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.534684 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.534689    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.534928 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.534699    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.534928 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.534710    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.596191 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.596159    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"]
Oct 30 19:13:21.603413 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.603386    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-predictor-75cc48778f-dfcmb"]
Oct 30 19:13:21.656261 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:21.656221    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" path="/var/lib/kubelet/pods/1f8094f5-a0d0-41ae-bc8b-3d59ede67e87/volumes"
Oct 30 19:13:24.883179 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.883151    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"]
Oct 30 19:13:24.883562 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.883426    4511 cpu_manager.go:414] "containerMap: removing container" podUID="1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" containerName="kube-rbac-proxy"
Oct 30 19:13:24.883562 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.883436    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" containerName="kube-rbac-proxy"
Oct 30 19:13:24.883562 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.883446    4511 cpu_manager.go:414] "containerMap: removing container" podUID="1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" containerName="kserve-container"
Oct 30 19:13:24.883562 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.883451    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" containerName="kserve-container"
Oct 30 19:13:24.883562 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.883461    4511 cpu_manager.go:414] "containerMap: removing container" podUID="1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" containerName="storage-initializer"
Oct 30 19:13:24.883562 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.883467    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" containerName="storage-initializer"
Oct 30 19:13:24.883562 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.883524    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" containerName="kube-rbac-proxy"
Oct 30 19:13:24.883562 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.883532    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f8094f5-a0d0-41ae-bc8b-3d59ede67e87" containerName="kserve-container"
Oct 30 19:13:24.886946 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.886926    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:24.937328 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.937302    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:13:24.937328 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.937311    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:13:24.957568 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.957542    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-864669f484-8j5dh\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:24.957682 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.957571    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-864669f484-8j5dh\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:24.957682 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.957605    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56k54\" (UniqueName: \"kubernetes.io/projected/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kube-api-access-56k54\") pod \"isvc-autogluon-v2-variants-predictor-864669f484-8j5dh\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:24.957807 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.957758    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-864669f484-8j5dh\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:24.957807 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.957798    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-864669f484-8j5dh\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:24.984665 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.984645    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Oct 30 19:13:24.984752 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.984706    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Oct 30 19:13:24.984752 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.984724    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:13:24.984899 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.984812    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:13:24.984899 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.984840    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:13:24.989539 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:24.989518    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"]
Oct 30 19:13:25.058728 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:25.058707    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-864669f484-8j5dh\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:25.058834 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:25.058736    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-864669f484-8j5dh\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:25.058834 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:25.058759    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-864669f484-8j5dh\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:25.058834 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:25.058778    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-864669f484-8j5dh\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:25.058834 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:25.058830    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56k54\" (UniqueName: \"kubernetes.io/projected/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kube-api-access-56k54\") pod \"isvc-autogluon-v2-variants-predictor-864669f484-8j5dh\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:25.059196 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:25.059176    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-864669f484-8j5dh\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:25.059472 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:25.059455    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-864669f484-8j5dh\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:25.059508 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:25.059463    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-864669f484-8j5dh\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:25.061270 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:25.061250    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-864669f484-8j5dh\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:25.117615 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:25.117587    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56k54\" (UniqueName: \"kubernetes.io/projected/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kube-api-access-56k54\") pod \"isvc-autogluon-v2-variants-predictor-864669f484-8j5dh\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:25.200150 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:25.200081    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:25.649023 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:25.648985    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"]
Oct 30 19:13:25.656541 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:13:25.656506    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c15bb97_e512_4a71_9a5f_c41159ffbe5d.slice/crio-c06d4286242a5ec0e5d94d5f0115d79c82a0c6ce5a733a6eed7d1541943f9729 WatchSource:0}: Error finding container c06d4286242a5ec0e5d94d5f0115d79c82a0c6ce5a733a6eed7d1541943f9729: Status 404 returned error can't find the container with id c06d4286242a5ec0e5d94d5f0115d79c82a0c6ce5a733a6eed7d1541943f9729
Oct 30 19:13:25.707383 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:25.707354    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:13:25.790858 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:25.790766    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 19:13:25.802048 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:25.797768    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 19:13:26.485993 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:26.485955    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh" event={"ID":"8c15bb97-e512-4a71-9a5f-c41159ffbe5d","Type":"ContainerStarted","Data":"7aeb57accd0a86f7ab319de6a60ed0a3ad9b1585c381c12bd09264a882ce3834"}
Oct 30 19:13:26.486412 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:26.486045    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh" event={"ID":"8c15bb97-e512-4a71-9a5f-c41159ffbe5d","Type":"ContainerStarted","Data":"c06d4286242a5ec0e5d94d5f0115d79c82a0c6ce5a733a6eed7d1541943f9729"}
Oct 30 19:13:32.507922 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:32.507886    4511 generic.go:356] "Generic (PLEG): container finished" podID="8c15bb97-e512-4a71-9a5f-c41159ffbe5d" containerID="7aeb57accd0a86f7ab319de6a60ed0a3ad9b1585c381c12bd09264a882ce3834" exitCode=0
Oct 30 19:13:32.508344 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:32.507958    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh" event={"ID":"8c15bb97-e512-4a71-9a5f-c41159ffbe5d","Type":"ContainerDied","Data":"7aeb57accd0a86f7ab319de6a60ed0a3ad9b1585c381c12bd09264a882ce3834"}
Oct 30 19:13:33.515350 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:33.515307    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh" event={"ID":"8c15bb97-e512-4a71-9a5f-c41159ffbe5d","Type":"ContainerStarted","Data":"8eb51123bef64fc58c3dae46006edfc2c45b1155e2cd32afb0e187f9ba906875"}
Oct 30 19:13:33.515861 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:33.515401    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh" event={"ID":"8c15bb97-e512-4a71-9a5f-c41159ffbe5d","Type":"ContainerStarted","Data":"a69679c520de699ff50203ee34792587bc6523f6026d476b26909dd455f92eb8"}
Oct 30 19:13:33.515861 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:33.515718    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:33.515861 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:33.515776    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:33.631082 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:33.631028    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh" podStartSLOduration=3.627547898 podStartE2EDuration="9.631016525s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.003468627s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:33.630757452 +0000 UTC m=+2108.148442352" watchObservedRunningTime="2026-10-30 19:13:33.631016525 +0000 UTC m=+2108.148701422"
Oct 30 19:13:39.526355 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:13:39.526277    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:13:40.156763 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:13:40.156727    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:14:10.156457 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:14:10.156417    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:14:40.158936 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:14:40.158897    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:15:08.684249 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:08.684171    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:15:10.156345 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:15:10.156307    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:15:19.693295 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:19.693246    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh" podUID="8c15bb97-e512-4a71-9a5f-c41159ffbe5d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.40:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:15:22.013270 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:22.013224    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"]
Oct 30 19:15:22.013758 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:22.013524    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh" podUID="8c15bb97-e512-4a71-9a5f-c41159ffbe5d" containerName="kserve-container" containerID="cri-o://a69679c520de699ff50203ee34792587bc6523f6026d476b26909dd455f92eb8" gracePeriod=30
Oct 30 19:15:22.013758 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:22.013553    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh" podUID="8c15bb97-e512-4a71-9a5f-c41159ffbe5d" containerName="kube-rbac-proxy" containerID="cri-o://8eb51123bef64fc58c3dae46006edfc2c45b1155e2cd32afb0e187f9ba906875" gracePeriod=30
Oct 30 19:15:22.914277 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:22.914242    4511 generic.go:356] "Generic (PLEG): container finished" podID="8c15bb97-e512-4a71-9a5f-c41159ffbe5d" containerID="8eb51123bef64fc58c3dae46006edfc2c45b1155e2cd32afb0e187f9ba906875" exitCode=0
Oct 30 19:15:22.914460 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:22.914314    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh" event={"ID":"8c15bb97-e512-4a71-9a5f-c41159ffbe5d","Type":"ContainerDied","Data":"8eb51123bef64fc58c3dae46006edfc2c45b1155e2cd32afb0e187f9ba906875"}
Oct 30 19:15:23.359177 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.359153    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:15:23.527779 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.527713    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-cabundle-cert\") pod \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") "
Oct 30 19:15:23.527911 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.527791    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kube-api-access-56k54\" (UniqueName: \"kubernetes.io/projected/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kube-api-access-56k54\") pod \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") "
Oct 30 19:15:23.527911 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.527839    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kserve-provision-location\") pod \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") "
Oct 30 19:15:23.527911 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.527870    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-proxy-tls\") pod \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") "
Oct 30 19:15:23.527911 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.527901    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kube-rbac-proxy-sar-config\") pod \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\" (UID: \"8c15bb97-e512-4a71-9a5f-c41159ffbe5d\") "
Oct 30 19:15:23.528196 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.528161    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-cabundle-cert" pod "8c15bb97-e512-4a71-9a5f-c41159ffbe5d" (UID: "8c15bb97-e512-4a71-9a5f-c41159ffbe5d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:23.528338 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.528287    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kube-rbac-proxy-sar-config" pod "8c15bb97-e512-4a71-9a5f-c41159ffbe5d" (UID: "8c15bb97-e512-4a71-9a5f-c41159ffbe5d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:23.530171 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.530090    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-proxy-tls" pod "8c15bb97-e512-4a71-9a5f-c41159ffbe5d" (UID: "8c15bb97-e512-4a71-9a5f-c41159ffbe5d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:23.530287 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.530267    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kube-api-access-56k54" pod "8c15bb97-e512-4a71-9a5f-c41159ffbe5d" (UID: "8c15bb97-e512-4a71-9a5f-c41159ffbe5d"). InnerVolumeSpecName "kube-api-access-56k54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:23.534991 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.534957    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kserve-provision-location" pod "8c15bb97-e512-4a71-9a5f-c41159ffbe5d" (UID: "8c15bb97-e512-4a71-9a5f-c41159ffbe5d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:23.628779 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.628757    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56k54\" (UniqueName: \"kubernetes.io/projected/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kube-api-access-56k54\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:23.628779 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.628778    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:23.628900 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.628789    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:23.628900 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.628798    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:23.628900 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.628807    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c15bb97-e512-4a71-9a5f-c41159ffbe5d-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:23.919392 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.919365    4511 generic.go:356] "Generic (PLEG): container finished" podID="8c15bb97-e512-4a71-9a5f-c41159ffbe5d" containerID="a69679c520de699ff50203ee34792587bc6523f6026d476b26909dd455f92eb8" exitCode=0
Oct 30 19:15:23.919521 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.919428    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"
Oct 30 19:15:23.919521 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.919443    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh" event={"ID":"8c15bb97-e512-4a71-9a5f-c41159ffbe5d","Type":"ContainerDied","Data":"a69679c520de699ff50203ee34792587bc6523f6026d476b26909dd455f92eb8"}
Oct 30 19:15:23.919521 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.919489    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh" event={"ID":"8c15bb97-e512-4a71-9a5f-c41159ffbe5d","Type":"ContainerDied","Data":"c06d4286242a5ec0e5d94d5f0115d79c82a0c6ce5a733a6eed7d1541943f9729"}
Oct 30 19:15:23.919521 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.919513    4511 scope.go:122] "RemoveContainer" containerID="8eb51123bef64fc58c3dae46006edfc2c45b1155e2cd32afb0e187f9ba906875"
Oct 30 19:15:23.927305 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.927289    4511 scope.go:122] "RemoveContainer" containerID="a69679c520de699ff50203ee34792587bc6523f6026d476b26909dd455f92eb8"
Oct 30 19:15:23.934445 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.934426    4511 scope.go:122] "RemoveContainer" containerID="7aeb57accd0a86f7ab319de6a60ed0a3ad9b1585c381c12bd09264a882ce3834"
Oct 30 19:15:23.941670 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.941651    4511 scope.go:122] "RemoveContainer" containerID="8eb51123bef64fc58c3dae46006edfc2c45b1155e2cd32afb0e187f9ba906875"
Oct 30 19:15:23.941919 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:15:23.941900    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb51123bef64fc58c3dae46006edfc2c45b1155e2cd32afb0e187f9ba906875\": container with ID starting with 8eb51123bef64fc58c3dae46006edfc2c45b1155e2cd32afb0e187f9ba906875 not found: ID does not exist" containerID="8eb51123bef64fc58c3dae46006edfc2c45b1155e2cd32afb0e187f9ba906875"
Oct 30 19:15:23.941979 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.941927    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb51123bef64fc58c3dae46006edfc2c45b1155e2cd32afb0e187f9ba906875"} err="failed to get container status \"8eb51123bef64fc58c3dae46006edfc2c45b1155e2cd32afb0e187f9ba906875\": rpc error: code = NotFound desc = could not find container \"8eb51123bef64fc58c3dae46006edfc2c45b1155e2cd32afb0e187f9ba906875\": container with ID starting with 8eb51123bef64fc58c3dae46006edfc2c45b1155e2cd32afb0e187f9ba906875 not found: ID does not exist"
Oct 30 19:15:23.941979 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.941946    4511 scope.go:122] "RemoveContainer" containerID="a69679c520de699ff50203ee34792587bc6523f6026d476b26909dd455f92eb8"
Oct 30 19:15:23.942202 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:15:23.942188    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a69679c520de699ff50203ee34792587bc6523f6026d476b26909dd455f92eb8\": container with ID starting with a69679c520de699ff50203ee34792587bc6523f6026d476b26909dd455f92eb8 not found: ID does not exist" containerID="a69679c520de699ff50203ee34792587bc6523f6026d476b26909dd455f92eb8"
Oct 30 19:15:23.942262 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.942206    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69679c520de699ff50203ee34792587bc6523f6026d476b26909dd455f92eb8"} err="failed to get container status \"a69679c520de699ff50203ee34792587bc6523f6026d476b26909dd455f92eb8\": rpc error: code = NotFound desc = could not find container \"a69679c520de699ff50203ee34792587bc6523f6026d476b26909dd455f92eb8\": container with ID starting with a69679c520de699ff50203ee34792587bc6523f6026d476b26909dd455f92eb8 not found: ID does not exist"
Oct 30 19:15:23.942262 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.942219    4511 scope.go:122] "RemoveContainer" containerID="7aeb57accd0a86f7ab319de6a60ed0a3ad9b1585c381c12bd09264a882ce3834"
Oct 30 19:15:23.942438 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:15:23.942423    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aeb57accd0a86f7ab319de6a60ed0a3ad9b1585c381c12bd09264a882ce3834\": container with ID starting with 7aeb57accd0a86f7ab319de6a60ed0a3ad9b1585c381c12bd09264a882ce3834 not found: ID does not exist" containerID="7aeb57accd0a86f7ab319de6a60ed0a3ad9b1585c381c12bd09264a882ce3834"
Oct 30 19:15:23.942476 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:23.942441    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aeb57accd0a86f7ab319de6a60ed0a3ad9b1585c381c12bd09264a882ce3834"} err="failed to get container status \"7aeb57accd0a86f7ab319de6a60ed0a3ad9b1585c381c12bd09264a882ce3834\": rpc error: code = NotFound desc = could not find container \"7aeb57accd0a86f7ab319de6a60ed0a3ad9b1585c381c12bd09264a882ce3834\": container with ID starting with 7aeb57accd0a86f7ab319de6a60ed0a3ad9b1585c381c12bd09264a882ce3834 not found: ID does not exist"
Oct 30 19:15:24.078287 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:24.078260    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"]
Oct 30 19:15:24.082365 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:24.082343    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-variants-predictor-864669f484-8j5dh"]
Oct 30 19:15:25.656093 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:25.656062    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c15bb97-e512-4a71-9a5f-c41159ffbe5d" path="/var/lib/kubelet/pods/8c15bb97-e512-4a71-9a5f-c41159ffbe5d/volumes"
Oct 30 19:15:27.048215 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.048184    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"]
Oct 30 19:15:27.048586 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.048469    4511 cpu_manager.go:414] "containerMap: removing container" podUID="8c15bb97-e512-4a71-9a5f-c41159ffbe5d" containerName="storage-initializer"
Oct 30 19:15:27.048586 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.048480    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c15bb97-e512-4a71-9a5f-c41159ffbe5d" containerName="storage-initializer"
Oct 30 19:15:27.048586 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.048490    4511 cpu_manager.go:414] "containerMap: removing container" podUID="8c15bb97-e512-4a71-9a5f-c41159ffbe5d" containerName="kserve-container"
Oct 30 19:15:27.048586 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.048495    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c15bb97-e512-4a71-9a5f-c41159ffbe5d" containerName="kserve-container"
Oct 30 19:15:27.048586 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.048509    4511 cpu_manager.go:414] "containerMap: removing container" podUID="8c15bb97-e512-4a71-9a5f-c41159ffbe5d" containerName="kube-rbac-proxy"
Oct 30 19:15:27.048586 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.048514    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c15bb97-e512-4a71-9a5f-c41159ffbe5d" containerName="kube-rbac-proxy"
Oct 30 19:15:27.048586 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.048565    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c15bb97-e512-4a71-9a5f-c41159ffbe5d" containerName="kserve-container"
Oct 30 19:15:27.048586 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.048573    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c15bb97-e512-4a71-9a5f-c41159ffbe5d" containerName="kube-rbac-proxy"
Oct 30 19:15:27.053303 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.053285    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:27.103661 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.103636    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:15:27.103768 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.103692    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:15:27.150916 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.150891    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:15:27.151187 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.151172    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Oct 30 19:15:27.151261 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.151178    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:15:27.151261 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.151223    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:15:27.151349 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.151292    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Oct 30 19:15:27.152513 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.152496    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:27.152559 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.152523    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:27.152559 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.152543    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:27.152626 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.152558    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:27.152626 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.152601    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmqkd\" (UniqueName: \"kubernetes.io/projected/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kube-api-access-gmqkd\") pod \"isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:27.154841 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.154817    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"]
Oct 30 19:15:27.253841 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.253815    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:27.253958 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.253847    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:27.253958 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.253874    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:27.253958 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.253897    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmqkd\" (UniqueName: \"kubernetes.io/projected/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kube-api-access-gmqkd\") pod \"isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:27.254118 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.253961    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:27.254388 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.254367    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:27.254584 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.254566    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:27.254642 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.254574    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:27.256451 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.256434    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:27.313651 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.313596    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmqkd\" (UniqueName: \"kubernetes.io/projected/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kube-api-access-gmqkd\") pod \"isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:27.370234 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.370213    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:27.792424 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.792350    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"]
Oct 30 19:15:27.799070 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:15:27.799041    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2615bd_a52a_4a24_98d5_f6a6b93978cc.slice/crio-d5fc211eda82822c1be3ac6fbea464912dee54ba7ca7e11f0c7178a05360e95a WatchSource:0}: Error finding container d5fc211eda82822c1be3ac6fbea464912dee54ba7ca7e11f0c7178a05360e95a: Status 404 returned error can't find the container with id d5fc211eda82822c1be3ac6fbea464912dee54ba7ca7e11f0c7178a05360e95a
Oct 30 19:15:27.936329 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.936292    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f" event={"ID":"5d2615bd-a52a-4a24-98d5-f6a6b93978cc","Type":"ContainerStarted","Data":"6c93853527fa80ef9dcbb6588dc572b044f8c78e338fc7cda6a120e4c699c353"}
Oct 30 19:15:27.936455 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:27.936360    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f" event={"ID":"5d2615bd-a52a-4a24-98d5-f6a6b93978cc","Type":"ContainerStarted","Data":"d5fc211eda82822c1be3ac6fbea464912dee54ba7ca7e11f0c7178a05360e95a"}
Oct 30 19:15:34.966071 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:34.966032    4511 generic.go:356] "Generic (PLEG): container finished" podID="5d2615bd-a52a-4a24-98d5-f6a6b93978cc" containerID="6c93853527fa80ef9dcbb6588dc572b044f8c78e338fc7cda6a120e4c699c353" exitCode=0
Oct 30 19:15:34.966629 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:34.966105    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f" event={"ID":"5d2615bd-a52a-4a24-98d5-f6a6b93978cc","Type":"ContainerDied","Data":"6c93853527fa80ef9dcbb6588dc572b044f8c78e338fc7cda6a120e4c699c353"}
Oct 30 19:15:35.972791 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:35.972756    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f" event={"ID":"5d2615bd-a52a-4a24-98d5-f6a6b93978cc","Type":"ContainerStarted","Data":"f7fcd7ecca0df5dd0015e56a9f49230982b2a5562e60e3ffe0dc9ad0ac9a3e69"}
Oct 30 19:15:35.973215 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:35.972818    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f" event={"ID":"5d2615bd-a52a-4a24-98d5-f6a6b93978cc","Type":"ContainerStarted","Data":"f75b03bf58db546a4f73dd7556df052b65b79813f6a48acc4cfc6a7d1547cabc"}
Oct 30 19:15:35.973215 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:35.973028    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:35.973215 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:35.973194    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:15:36.136143 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:36.136077    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f" podStartSLOduration=2.953887632 podStartE2EDuration="9.136063056s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.182175424s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:36.135338172 +0000 UTC m=+2230.653023073" watchObservedRunningTime="2026-10-30 19:15:36.136063056 +0000 UTC m=+2230.653747955"
Oct 30 19:15:40.157903 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:15:40.157864    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:15:41.982260 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:15:41.982234    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:16:10.155660 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:16:10.155625    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:16:40.156235 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:16:40.156147    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:17:10.156565 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:17:10.156520    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:17:14.683148 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:14.683094    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:17:33.574404 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:33.574372    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"]
Oct 30 19:17:33.574932 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:33.574603    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f" podUID="5d2615bd-a52a-4a24-98d5-f6a6b93978cc" containerName="kserve-container" containerID="cri-o://f75b03bf58db546a4f73dd7556df052b65b79813f6a48acc4cfc6a7d1547cabc" gracePeriod=30
Oct 30 19:17:33.574932 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:33.574645    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f" podUID="5d2615bd-a52a-4a24-98d5-f6a6b93978cc" containerName="kube-rbac-proxy" containerID="cri-o://f7fcd7ecca0df5dd0015e56a9f49230982b2a5562e60e3ffe0dc9ad0ac9a3e69" gracePeriod=30
Oct 30 19:17:34.408999 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:34.408962    4511 generic.go:356] "Generic (PLEG): container finished" podID="5d2615bd-a52a-4a24-98d5-f6a6b93978cc" containerID="f7fcd7ecca0df5dd0015e56a9f49230982b2a5562e60e3ffe0dc9ad0ac9a3e69" exitCode=0
Oct 30 19:17:34.409187 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:34.409034    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f" event={"ID":"5d2615bd-a52a-4a24-98d5-f6a6b93978cc","Type":"ContainerDied","Data":"f7fcd7ecca0df5dd0015e56a9f49230982b2a5562e60e3ffe0dc9ad0ac9a3e69"}
Oct 30 19:17:34.651952 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:34.651909    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f" podUID="5d2615bd-a52a-4a24-98d5-f6a6b93978cc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.41:8080/v2/models/isvc-autogluon-v2-noslash/ready\": dial tcp 10.133.0.41:8080: connect: connection refused"
Oct 30 19:17:34.921442 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:34.921420    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:17:34.989887 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:34.989864    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kserve-provision-location\") pod \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") "
Oct 30 19:17:34.989997 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:34.989892    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-cabundle-cert\") pod \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") "
Oct 30 19:17:34.989997 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:34.989931    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-proxy-tls\") pod \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") "
Oct 30 19:17:34.989997 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:34.989960    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kube-rbac-proxy-sar-config\") pod \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") "
Oct 30 19:17:34.989997 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:34.989990    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kube-api-access-gmqkd\" (UniqueName: \"kubernetes.io/projected/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kube-api-access-gmqkd\") pod \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\" (UID: \"5d2615bd-a52a-4a24-98d5-f6a6b93978cc\") "
Oct 30 19:17:34.990327 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:34.990299    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-cabundle-cert" pod "5d2615bd-a52a-4a24-98d5-f6a6b93978cc" (UID: "5d2615bd-a52a-4a24-98d5-f6a6b93978cc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:34.990414 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:34.990383    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kube-rbac-proxy-sar-config" pod "5d2615bd-a52a-4a24-98d5-f6a6b93978cc" (UID: "5d2615bd-a52a-4a24-98d5-f6a6b93978cc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:34.992237 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:34.992208    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-proxy-tls" pod "5d2615bd-a52a-4a24-98d5-f6a6b93978cc" (UID: "5d2615bd-a52a-4a24-98d5-f6a6b93978cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:34.992320 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:34.992228    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kube-api-access-gmqkd" pod "5d2615bd-a52a-4a24-98d5-f6a6b93978cc" (UID: "5d2615bd-a52a-4a24-98d5-f6a6b93978cc"). InnerVolumeSpecName "kube-api-access-gmqkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:34.994498 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:34.994474    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kserve-provision-location" pod "5d2615bd-a52a-4a24-98d5-f6a6b93978cc" (UID: "5d2615bd-a52a-4a24-98d5-f6a6b93978cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:35.090682 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.090654    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:35.090801 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.090686    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:35.090801 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.090702    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:35.090801 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.090716    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:35.090801 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.090730    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmqkd\" (UniqueName: \"kubernetes.io/projected/5d2615bd-a52a-4a24-98d5-f6a6b93978cc-kube-api-access-gmqkd\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:35.414279 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.414198    4511 generic.go:356] "Generic (PLEG): container finished" podID="5d2615bd-a52a-4a24-98d5-f6a6b93978cc" containerID="f75b03bf58db546a4f73dd7556df052b65b79813f6a48acc4cfc6a7d1547cabc" exitCode=0
Oct 30 19:17:35.414279 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.414260    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"
Oct 30 19:17:35.414451 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.414251    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f" event={"ID":"5d2615bd-a52a-4a24-98d5-f6a6b93978cc","Type":"ContainerDied","Data":"f75b03bf58db546a4f73dd7556df052b65b79813f6a48acc4cfc6a7d1547cabc"}
Oct 30 19:17:35.414451 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.414405    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f" event={"ID":"5d2615bd-a52a-4a24-98d5-f6a6b93978cc","Type":"ContainerDied","Data":"d5fc211eda82822c1be3ac6fbea464912dee54ba7ca7e11f0c7178a05360e95a"}
Oct 30 19:17:35.414451 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.414432    4511 scope.go:122] "RemoveContainer" containerID="f7fcd7ecca0df5dd0015e56a9f49230982b2a5562e60e3ffe0dc9ad0ac9a3e69"
Oct 30 19:17:35.423038 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.423016    4511 scope.go:122] "RemoveContainer" containerID="f75b03bf58db546a4f73dd7556df052b65b79813f6a48acc4cfc6a7d1547cabc"
Oct 30 19:17:35.430178 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.430159    4511 scope.go:122] "RemoveContainer" containerID="6c93853527fa80ef9dcbb6588dc572b044f8c78e338fc7cda6a120e4c699c353"
Oct 30 19:17:35.437039 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.437023    4511 scope.go:122] "RemoveContainer" containerID="f7fcd7ecca0df5dd0015e56a9f49230982b2a5562e60e3ffe0dc9ad0ac9a3e69"
Oct 30 19:17:35.437283 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:17:35.437264    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7fcd7ecca0df5dd0015e56a9f49230982b2a5562e60e3ffe0dc9ad0ac9a3e69\": container with ID starting with f7fcd7ecca0df5dd0015e56a9f49230982b2a5562e60e3ffe0dc9ad0ac9a3e69 not found: ID does not exist" containerID="f7fcd7ecca0df5dd0015e56a9f49230982b2a5562e60e3ffe0dc9ad0ac9a3e69"
Oct 30 19:17:35.437357 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.437289    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7fcd7ecca0df5dd0015e56a9f49230982b2a5562e60e3ffe0dc9ad0ac9a3e69"} err="failed to get container status \"f7fcd7ecca0df5dd0015e56a9f49230982b2a5562e60e3ffe0dc9ad0ac9a3e69\": rpc error: code = NotFound desc = could not find container \"f7fcd7ecca0df5dd0015e56a9f49230982b2a5562e60e3ffe0dc9ad0ac9a3e69\": container with ID starting with f7fcd7ecca0df5dd0015e56a9f49230982b2a5562e60e3ffe0dc9ad0ac9a3e69 not found: ID does not exist"
Oct 30 19:17:35.437357 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.437306    4511 scope.go:122] "RemoveContainer" containerID="f75b03bf58db546a4f73dd7556df052b65b79813f6a48acc4cfc6a7d1547cabc"
Oct 30 19:17:35.437529 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:17:35.437515    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f75b03bf58db546a4f73dd7556df052b65b79813f6a48acc4cfc6a7d1547cabc\": container with ID starting with f75b03bf58db546a4f73dd7556df052b65b79813f6a48acc4cfc6a7d1547cabc not found: ID does not exist" containerID="f75b03bf58db546a4f73dd7556df052b65b79813f6a48acc4cfc6a7d1547cabc"
Oct 30 19:17:35.437568 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.437533    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75b03bf58db546a4f73dd7556df052b65b79813f6a48acc4cfc6a7d1547cabc"} err="failed to get container status \"f75b03bf58db546a4f73dd7556df052b65b79813f6a48acc4cfc6a7d1547cabc\": rpc error: code = NotFound desc = could not find container \"f75b03bf58db546a4f73dd7556df052b65b79813f6a48acc4cfc6a7d1547cabc\": container with ID starting with f75b03bf58db546a4f73dd7556df052b65b79813f6a48acc4cfc6a7d1547cabc not found: ID does not exist"
Oct 30 19:17:35.437568 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.437546    4511 scope.go:122] "RemoveContainer" containerID="6c93853527fa80ef9dcbb6588dc572b044f8c78e338fc7cda6a120e4c699c353"
Oct 30 19:17:35.437769 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:17:35.437751    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c93853527fa80ef9dcbb6588dc572b044f8c78e338fc7cda6a120e4c699c353\": container with ID starting with 6c93853527fa80ef9dcbb6588dc572b044f8c78e338fc7cda6a120e4c699c353 not found: ID does not exist" containerID="6c93853527fa80ef9dcbb6588dc572b044f8c78e338fc7cda6a120e4c699c353"
Oct 30 19:17:35.437812 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.437771    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c93853527fa80ef9dcbb6588dc572b044f8c78e338fc7cda6a120e4c699c353"} err="failed to get container status \"6c93853527fa80ef9dcbb6588dc572b044f8c78e338fc7cda6a120e4c699c353\": rpc error: code = NotFound desc = could not find container \"6c93853527fa80ef9dcbb6588dc572b044f8c78e338fc7cda6a120e4c699c353\": container with ID starting with 6c93853527fa80ef9dcbb6588dc572b044f8c78e338fc7cda6a120e4c699c353 not found: ID does not exist"
Oct 30 19:17:35.572858 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.572833    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"]
Oct 30 19:17:35.581806 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.581783    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-v2-noslash-predictor-7b488cdb5c-qkt4f"]
Oct 30 19:17:35.655274 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:35.655254    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d2615bd-a52a-4a24-98d5-f6a6b93978cc" path="/var/lib/kubelet/pods/5d2615bd-a52a-4a24-98d5-f6a6b93978cc/volumes"
Oct 30 19:17:38.667323 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.667292    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"]
Oct 30 19:17:38.667765 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.667568    4511 cpu_manager.go:414] "containerMap: removing container" podUID="5d2615bd-a52a-4a24-98d5-f6a6b93978cc" containerName="kube-rbac-proxy"
Oct 30 19:17:38.667765 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.667578    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2615bd-a52a-4a24-98d5-f6a6b93978cc" containerName="kube-rbac-proxy"
Oct 30 19:17:38.667765 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.667589    4511 cpu_manager.go:414] "containerMap: removing container" podUID="5d2615bd-a52a-4a24-98d5-f6a6b93978cc" containerName="storage-initializer"
Oct 30 19:17:38.667765 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.667594    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2615bd-a52a-4a24-98d5-f6a6b93978cc" containerName="storage-initializer"
Oct 30 19:17:38.667765 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.667611    4511 cpu_manager.go:414] "containerMap: removing container" podUID="5d2615bd-a52a-4a24-98d5-f6a6b93978cc" containerName="kserve-container"
Oct 30 19:17:38.667765 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.667616    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2615bd-a52a-4a24-98d5-f6a6b93978cc" containerName="kserve-container"
Oct 30 19:17:38.667765 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.667656    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d2615bd-a52a-4a24-98d5-f6a6b93978cc" containerName="kube-rbac-proxy"
Oct 30 19:17:38.667765 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.667667    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d2615bd-a52a-4a24-98d5-f6a6b93978cc" containerName="kserve-container"
Oct 30 19:17:38.672116 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.672096    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:38.719008 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.718978    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d911057-6576-4751-a912-37d5691ac6b7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:38.719164 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.719015    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kjzc\" (UniqueName: \"kubernetes.io/projected/3d911057-6576-4751-a912-37d5691ac6b7-kube-api-access-4kjzc\") pod \"isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:38.719164 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.719036    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d911057-6576-4751-a912-37d5691ac6b7-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:38.719164 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.719090    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d911057-6576-4751-a912-37d5691ac6b7-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:38.719324 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.719204    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d911057-6576-4751-a912-37d5691ac6b7-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:38.722929 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.722909    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:17:38.770066 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.770045    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Oct 30 19:17:38.770374 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.770338    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Oct 30 19:17:38.770374 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.770360    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:17:38.770512 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.770378    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:17:38.770512 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.770403    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:17:38.770512 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.770452    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:17:38.771749 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.771732    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"]
Oct 30 19:17:38.819728 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.819699    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d911057-6576-4751-a912-37d5691ac6b7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:38.819898 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.819734    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kjzc\" (UniqueName: \"kubernetes.io/projected/3d911057-6576-4751-a912-37d5691ac6b7-kube-api-access-4kjzc\") pod \"isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:38.819898 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.819752    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d911057-6576-4751-a912-37d5691ac6b7-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:38.819898 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.819775    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d911057-6576-4751-a912-37d5691ac6b7-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:38.819898 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.819801    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d911057-6576-4751-a912-37d5691ac6b7-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:38.820282 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.820262    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d911057-6576-4751-a912-37d5691ac6b7-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:38.820569 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.820513    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d911057-6576-4751-a912-37d5691ac6b7-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:38.820569 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.820513    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d911057-6576-4751-a912-37d5691ac6b7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:38.822278 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.822259    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d911057-6576-4751-a912-37d5691ac6b7-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:38.878770 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.878739    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kjzc\" (UniqueName: \"kubernetes.io/projected/3d911057-6576-4751-a912-37d5691ac6b7-kube-api-access-4kjzc\") pod \"isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:38.988362 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:38.988276    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:39.442429 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:39.442400    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"]
Oct 30 19:17:39.449167 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:17:39.449117    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d911057_6576_4751_a912_37d5691ac6b7.slice/crio-e5bd651f959c496db19a8bd3417ced4626aa7bba8773da6755b3daad108e13fa WatchSource:0}: Error finding container e5bd651f959c496db19a8bd3417ced4626aa7bba8773da6755b3daad108e13fa: Status 404 returned error can't find the container with id e5bd651f959c496db19a8bd3417ced4626aa7bba8773da6755b3daad108e13fa
Oct 30 19:17:40.156984 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:17:40.156946    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:17:40.435779 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:40.435690    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv" event={"ID":"3d911057-6576-4751-a912-37d5691ac6b7","Type":"ContainerStarted","Data":"d53e6d735e54c6082a29609752533917a33d9d5901fa65ba305dde1d124c4ea5"}
Oct 30 19:17:40.435950 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:40.435799    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv" event={"ID":"3d911057-6576-4751-a912-37d5691ac6b7","Type":"ContainerStarted","Data":"e5bd651f959c496db19a8bd3417ced4626aa7bba8773da6755b3daad108e13fa"}
Oct 30 19:17:51.478497 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:51.478466    4511 generic.go:356] "Generic (PLEG): container finished" podID="3d911057-6576-4751-a912-37d5691ac6b7" containerID="d53e6d735e54c6082a29609752533917a33d9d5901fa65ba305dde1d124c4ea5" exitCode=0
Oct 30 19:17:51.478497 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:51.478502    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv" event={"ID":"3d911057-6576-4751-a912-37d5691ac6b7","Type":"ContainerDied","Data":"d53e6d735e54c6082a29609752533917a33d9d5901fa65ba305dde1d124c4ea5"}
Oct 30 19:17:52.485800 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:52.485767    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv" event={"ID":"3d911057-6576-4751-a912-37d5691ac6b7","Type":"ContainerStarted","Data":"ea13f9331cf80513ad402ac669d1948a642f5041a12550c480001c622282564c"}
Oct 30 19:17:52.486226 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:52.485826    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv" event={"ID":"3d911057-6576-4751-a912-37d5691ac6b7","Type":"ContainerStarted","Data":"33d740ac6da9a958e9506d0b84fb260c7b44095cefaa57279caf3ddae6f54ecc"}
Oct 30 19:17:52.486226 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:52.486023    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:52.603171 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:52.603103    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv" podStartSLOduration=3.446255344 podStartE2EDuration="14.603089554s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.15683421s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:52.602253286 +0000 UTC m=+2367.119938188" watchObservedRunningTime="2026-10-30 19:17:52.603089554 +0000 UTC m=+2367.120774507"
Oct 30 19:17:53.491067 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:53.491032    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:53.492403 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:53.492372    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv" podUID="3d911057-6576-4751-a912-37d5691ac6b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Oct 30 19:17:54.494944 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:54.494897    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv" podUID="3d911057-6576-4751-a912-37d5691ac6b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Oct 30 19:17:59.499255 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:59.499226    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:17:59.499770 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:17:59.499695    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv" podUID="3d911057-6576-4751-a912-37d5691ac6b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Oct 30 19:18:09.501081 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:09.501053    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:18:10.156805 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:18:10.156766    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:18:25.814628 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:25.814599    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 19:18:25.822968 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:25.822950    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 19:18:30.755445 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:30.755413    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"]
Oct 30 19:18:30.755826 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:30.755639    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv" podUID="3d911057-6576-4751-a912-37d5691ac6b7" containerName="kserve-container" containerID="cri-o://33d740ac6da9a958e9506d0b84fb260c7b44095cefaa57279caf3ddae6f54ecc" gracePeriod=30
Oct 30 19:18:30.755826 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:30.755720    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv" podUID="3d911057-6576-4751-a912-37d5691ac6b7" containerName="kube-rbac-proxy" containerID="cri-o://ea13f9331cf80513ad402ac669d1948a642f5041a12550c480001c622282564c" gracePeriod=30
Oct 30 19:18:31.640268 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:31.640234    4511 generic.go:356] "Generic (PLEG): container finished" podID="3d911057-6576-4751-a912-37d5691ac6b7" containerID="ea13f9331cf80513ad402ac669d1948a642f5041a12550c480001c622282564c" exitCode=0
Oct 30 19:18:31.640472 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:31.640288    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv" event={"ID":"3d911057-6576-4751-a912-37d5691ac6b7","Type":"ContainerDied","Data":"ea13f9331cf80513ad402ac669d1948a642f5041a12550c480001c622282564c"}
Oct 30 19:18:32.397507 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.397484    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:18:32.540225 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.540161    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d911057-6576-4751-a912-37d5691ac6b7-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d911057-6576-4751-a912-37d5691ac6b7-cabundle-cert\") pod \"3d911057-6576-4751-a912-37d5691ac6b7\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") "
Oct 30 19:18:32.540225 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.540205    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d911057-6576-4751-a912-37d5691ac6b7-kube-api-access-4kjzc\" (UniqueName: \"kubernetes.io/projected/3d911057-6576-4751-a912-37d5691ac6b7-kube-api-access-4kjzc\") pod \"3d911057-6576-4751-a912-37d5691ac6b7\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") "
Oct 30 19:18:32.540225 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.540224    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d911057-6576-4751-a912-37d5691ac6b7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d911057-6576-4751-a912-37d5691ac6b7-kube-rbac-proxy-sar-config\") pod \"3d911057-6576-4751-a912-37d5691ac6b7\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") "
Oct 30 19:18:32.540431 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.540240    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d911057-6576-4751-a912-37d5691ac6b7-proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d911057-6576-4751-a912-37d5691ac6b7-proxy-tls\") pod \"3d911057-6576-4751-a912-37d5691ac6b7\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") "
Oct 30 19:18:32.540431 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.540317    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d911057-6576-4751-a912-37d5691ac6b7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d911057-6576-4751-a912-37d5691ac6b7-kserve-provision-location\") pod \"3d911057-6576-4751-a912-37d5691ac6b7\" (UID: \"3d911057-6576-4751-a912-37d5691ac6b7\") "
Oct 30 19:18:32.540684 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.540657    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d911057-6576-4751-a912-37d5691ac6b7-cabundle-cert" pod "3d911057-6576-4751-a912-37d5691ac6b7" (UID: "3d911057-6576-4751-a912-37d5691ac6b7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:32.540771 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.540678    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d911057-6576-4751-a912-37d5691ac6b7-kube-rbac-proxy-sar-config" pod "3d911057-6576-4751-a912-37d5691ac6b7" (UID: "3d911057-6576-4751-a912-37d5691ac6b7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:32.541240 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.541221    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d911057-6576-4751-a912-37d5691ac6b7-kserve-provision-location" pod "3d911057-6576-4751-a912-37d5691ac6b7" (UID: "3d911057-6576-4751-a912-37d5691ac6b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:32.542493 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.542472    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d911057-6576-4751-a912-37d5691ac6b7-proxy-tls" pod "3d911057-6576-4751-a912-37d5691ac6b7" (UID: "3d911057-6576-4751-a912-37d5691ac6b7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:32.542589 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.542495    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d911057-6576-4751-a912-37d5691ac6b7-kube-api-access-4kjzc" pod "3d911057-6576-4751-a912-37d5691ac6b7" (UID: "3d911057-6576-4751-a912-37d5691ac6b7"). InnerVolumeSpecName "kube-api-access-4kjzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:32.641680 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.641646    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d911057-6576-4751-a912-37d5691ac6b7-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:32.641680 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.641681    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d911057-6576-4751-a912-37d5691ac6b7-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:32.641855 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.641696    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kjzc\" (UniqueName: \"kubernetes.io/projected/3d911057-6576-4751-a912-37d5691ac6b7-kube-api-access-4kjzc\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:32.641855 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.641711    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d911057-6576-4751-a912-37d5691ac6b7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:32.641855 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.641725    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d911057-6576-4751-a912-37d5691ac6b7-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:32.646794 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.646766    4511 generic.go:356] "Generic (PLEG): container finished" podID="3d911057-6576-4751-a912-37d5691ac6b7" containerID="33d740ac6da9a958e9506d0b84fb260c7b44095cefaa57279caf3ddae6f54ecc" exitCode=0
Oct 30 19:18:32.646940 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.646823    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"
Oct 30 19:18:32.646940 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.646820    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv" event={"ID":"3d911057-6576-4751-a912-37d5691ac6b7","Type":"ContainerDied","Data":"33d740ac6da9a958e9506d0b84fb260c7b44095cefaa57279caf3ddae6f54ecc"}
Oct 30 19:18:32.647061 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.646966    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv" event={"ID":"3d911057-6576-4751-a912-37d5691ac6b7","Type":"ContainerDied","Data":"e5bd651f959c496db19a8bd3417ced4626aa7bba8773da6755b3daad108e13fa"}
Oct 30 19:18:32.647061 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.647009    4511 scope.go:122] "RemoveContainer" containerID="ea13f9331cf80513ad402ac669d1948a642f5041a12550c480001c622282564c"
Oct 30 19:18:32.655917 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.655903    4511 scope.go:122] "RemoveContainer" containerID="33d740ac6da9a958e9506d0b84fb260c7b44095cefaa57279caf3ddae6f54ecc"
Oct 30 19:18:32.663072 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.663050    4511 scope.go:122] "RemoveContainer" containerID="d53e6d735e54c6082a29609752533917a33d9d5901fa65ba305dde1d124c4ea5"
Oct 30 19:18:32.669999 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.669983    4511 scope.go:122] "RemoveContainer" containerID="ea13f9331cf80513ad402ac669d1948a642f5041a12550c480001c622282564c"
Oct 30 19:18:32.670266 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:18:32.670252    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea13f9331cf80513ad402ac669d1948a642f5041a12550c480001c622282564c\": container with ID starting with ea13f9331cf80513ad402ac669d1948a642f5041a12550c480001c622282564c not found: ID does not exist" containerID="ea13f9331cf80513ad402ac669d1948a642f5041a12550c480001c622282564c"
Oct 30 19:18:32.670326 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.670273    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea13f9331cf80513ad402ac669d1948a642f5041a12550c480001c622282564c"} err="failed to get container status \"ea13f9331cf80513ad402ac669d1948a642f5041a12550c480001c622282564c\": rpc error: code = NotFound desc = could not find container \"ea13f9331cf80513ad402ac669d1948a642f5041a12550c480001c622282564c\": container with ID starting with ea13f9331cf80513ad402ac669d1948a642f5041a12550c480001c622282564c not found: ID does not exist"
Oct 30 19:18:32.670326 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.670287    4511 scope.go:122] "RemoveContainer" containerID="33d740ac6da9a958e9506d0b84fb260c7b44095cefaa57279caf3ddae6f54ecc"
Oct 30 19:18:32.670505 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:18:32.670490    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33d740ac6da9a958e9506d0b84fb260c7b44095cefaa57279caf3ddae6f54ecc\": container with ID starting with 33d740ac6da9a958e9506d0b84fb260c7b44095cefaa57279caf3ddae6f54ecc not found: ID does not exist" containerID="33d740ac6da9a958e9506d0b84fb260c7b44095cefaa57279caf3ddae6f54ecc"
Oct 30 19:18:32.670549 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.670512    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33d740ac6da9a958e9506d0b84fb260c7b44095cefaa57279caf3ddae6f54ecc"} err="failed to get container status \"33d740ac6da9a958e9506d0b84fb260c7b44095cefaa57279caf3ddae6f54ecc\": rpc error: code = NotFound desc = could not find container \"33d740ac6da9a958e9506d0b84fb260c7b44095cefaa57279caf3ddae6f54ecc\": container with ID starting with 33d740ac6da9a958e9506d0b84fb260c7b44095cefaa57279caf3ddae6f54ecc not found: ID does not exist"
Oct 30 19:18:32.670549 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.670524    4511 scope.go:122] "RemoveContainer" containerID="d53e6d735e54c6082a29609752533917a33d9d5901fa65ba305dde1d124c4ea5"
Oct 30 19:18:32.670735 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:18:32.670716    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d53e6d735e54c6082a29609752533917a33d9d5901fa65ba305dde1d124c4ea5\": container with ID starting with d53e6d735e54c6082a29609752533917a33d9d5901fa65ba305dde1d124c4ea5 not found: ID does not exist" containerID="d53e6d735e54c6082a29609752533917a33d9d5901fa65ba305dde1d124c4ea5"
Oct 30 19:18:32.670774 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.670739    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d53e6d735e54c6082a29609752533917a33d9d5901fa65ba305dde1d124c4ea5"} err="failed to get container status \"d53e6d735e54c6082a29609752533917a33d9d5901fa65ba305dde1d124c4ea5\": rpc error: code = NotFound desc = could not find container \"d53e6d735e54c6082a29609752533917a33d9d5901fa65ba305dde1d124c4ea5\": container with ID starting with d53e6d735e54c6082a29609752533917a33d9d5901fa65ba305dde1d124c4ea5 not found: ID does not exist"
Oct 30 19:18:32.805225 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.805198    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"]
Oct 30 19:18:32.809387 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:32.809367    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-predictor-5c76cf8d4-csfsv"]
Oct 30 19:18:33.655450 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:33.655416    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d911057-6576-4751-a912-37d5691ac6b7" path="/var/lib/kubelet/pods/3d911057-6576-4751-a912-37d5691ac6b7/volumes"
Oct 30 19:18:35.861810 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.861772    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"]
Oct 30 19:18:35.862223 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.862078    4511 cpu_manager.go:414] "containerMap: removing container" podUID="3d911057-6576-4751-a912-37d5691ac6b7" containerName="storage-initializer"
Oct 30 19:18:35.862223 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.862090    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d911057-6576-4751-a912-37d5691ac6b7" containerName="storage-initializer"
Oct 30 19:18:35.862223 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.862117    4511 cpu_manager.go:414] "containerMap: removing container" podUID="3d911057-6576-4751-a912-37d5691ac6b7" containerName="kserve-container"
Oct 30 19:18:35.862223 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.862123    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d911057-6576-4751-a912-37d5691ac6b7" containerName="kserve-container"
Oct 30 19:18:35.862223 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.862145    4511 cpu_manager.go:414] "containerMap: removing container" podUID="3d911057-6576-4751-a912-37d5691ac6b7" containerName="kube-rbac-proxy"
Oct 30 19:18:35.862223 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.862150    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d911057-6576-4751-a912-37d5691ac6b7" containerName="kube-rbac-proxy"
Oct 30 19:18:35.862223 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.862204    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d911057-6576-4751-a912-37d5691ac6b7" containerName="kserve-container"
Oct 30 19:18:35.862223 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.862213    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d911057-6576-4751-a912-37d5691ac6b7" containerName="kube-rbac-proxy"
Oct 30 19:18:35.867007 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.866986    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:35.917625 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.917595    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:18:35.917625 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.917624    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:18:35.964998 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.964971    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:18:35.964998 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.964971    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Oct 30 19:18:35.965203 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.965192    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:18:35.965257 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.965211    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:18:35.965309 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.965280    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Oct 30 19:18:35.965725 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.965694    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30879961-3956-4ffb-b399-7c406c65e0c8-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:35.965807 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.965784    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2xm5\" (UniqueName: \"kubernetes.io/projected/30879961-3956-4ffb-b399-7c406c65e0c8-kube-api-access-d2xm5\") pod \"isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:35.965872 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.965830    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30879961-3956-4ffb-b399-7c406c65e0c8-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:35.965930 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.965913    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30879961-3956-4ffb-b399-7c406c65e0c8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:35.965988 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.965970    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30879961-3956-4ffb-b399-7c406c65e0c8-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:35.968444 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:35.968422    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"]
Oct 30 19:18:36.066435 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:36.066411    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30879961-3956-4ffb-b399-7c406c65e0c8-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:36.066579 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:36.066448    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30879961-3956-4ffb-b399-7c406c65e0c8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:36.066579 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:36.066493    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30879961-3956-4ffb-b399-7c406c65e0c8-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:36.066579 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:36.066526    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30879961-3956-4ffb-b399-7c406c65e0c8-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:36.066579 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:36.066550    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xm5\" (UniqueName: \"kubernetes.io/projected/30879961-3956-4ffb-b399-7c406c65e0c8-kube-api-access-d2xm5\") pod \"isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:36.066945 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:36.066908    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30879961-3956-4ffb-b399-7c406c65e0c8-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:36.067270 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:36.067250    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30879961-3956-4ffb-b399-7c406c65e0c8-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:36.067466 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:36.067446    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30879961-3956-4ffb-b399-7c406c65e0c8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:36.068976 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:36.068949    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30879961-3956-4ffb-b399-7c406c65e0c8-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:36.125680 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:36.125627    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xm5\" (UniqueName: \"kubernetes.io/projected/30879961-3956-4ffb-b399-7c406c65e0c8-kube-api-access-d2xm5\") pod \"isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") " pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:36.183621 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:36.183596    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:36.604210 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:36.604175    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"]
Oct 30 19:18:36.611886 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:18:36.611855    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30879961_3956_4ffb_b399_7c406c65e0c8.slice/crio-5abb3d480c0e6d64aa3414f1198f3646a9067818e5c479d56a4f64c8646401d8 WatchSource:0}: Error finding container 5abb3d480c0e6d64aa3414f1198f3646a9067818e5c479d56a4f64c8646401d8: Status 404 returned error can't find the container with id 5abb3d480c0e6d64aa3414f1198f3646a9067818e5c479d56a4f64c8646401d8
Oct 30 19:18:36.666816 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:36.666792    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5" event={"ID":"30879961-3956-4ffb-b399-7c406c65e0c8","Type":"ContainerStarted","Data":"5abb3d480c0e6d64aa3414f1198f3646a9067818e5c479d56a4f64c8646401d8"}
Oct 30 19:18:37.671218 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:37.671182    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5" event={"ID":"30879961-3956-4ffb-b399-7c406c65e0c8","Type":"ContainerStarted","Data":"faf7d714c7407d4747f4a514d179bef05d99a12c0fe9fc648f8a44663b574a3a"}
Oct 30 19:18:40.156675 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:18:40.156636    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:18:42.691981 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:42.691947    4511 generic.go:356] "Generic (PLEG): container finished" podID="30879961-3956-4ffb-b399-7c406c65e0c8" containerID="faf7d714c7407d4747f4a514d179bef05d99a12c0fe9fc648f8a44663b574a3a" exitCode=0
Oct 30 19:18:42.692441 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:42.692020    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5" event={"ID":"30879961-3956-4ffb-b399-7c406c65e0c8","Type":"ContainerDied","Data":"faf7d714c7407d4747f4a514d179bef05d99a12c0fe9fc648f8a44663b574a3a"}
Oct 30 19:18:43.698877 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:43.698842    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5" event={"ID":"30879961-3956-4ffb-b399-7c406c65e0c8","Type":"ContainerStarted","Data":"4a296a81b2886288352aa29970547993a833056f5e8d745af4ebcd351ecbd0c9"}
Oct 30 19:18:43.699415 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:43.698905    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5" event={"ID":"30879961-3956-4ffb-b399-7c406c65e0c8","Type":"ContainerStarted","Data":"64f985d1aaf92eddb3f5ddaf4651fb271957644a6cc9f4971c0421244e9fc04f"}
Oct 30 19:18:43.699415 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:43.699140    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:43.831932 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:43.831870    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5" podStartSLOduration=3.574150522 podStartE2EDuration="8.831857018s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.257706496s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:43.830603298 +0000 UTC m=+2418.348288197" watchObservedRunningTime="2026-10-30 19:18:43.831857018 +0000 UTC m=+2418.349541979"
Oct 30 19:18:44.703190 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:44.703157    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:44.704447 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:44.704407    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5" podUID="30879961-3956-4ffb-b399-7c406c65e0c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Oct 30 19:18:45.707468 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:45.707429    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5" podUID="30879961-3956-4ffb-b399-7c406c65e0c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Oct 30 19:18:50.712875 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:50.712849    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:18:50.713330 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:18:50.713305    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5" podUID="30879961-3956-4ffb-b399-7c406c65e0c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Oct 30 19:19:00.714165 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:00.714117    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:19:10.154961 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:19:10.154924    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:19:11.164078 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:11.164036    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"]
Oct 30 19:19:11.164575 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:11.164348    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5" podUID="30879961-3956-4ffb-b399-7c406c65e0c8" containerName="kserve-container" containerID="cri-o://64f985d1aaf92eddb3f5ddaf4651fb271957644a6cc9f4971c0421244e9fc04f" gracePeriod=30
Oct 30 19:19:11.164575 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:11.164388    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5" podUID="30879961-3956-4ffb-b399-7c406c65e0c8" containerName="kube-rbac-proxy" containerID="cri-o://4a296a81b2886288352aa29970547993a833056f5e8d745af4ebcd351ecbd0c9" gracePeriod=30
Oct 30 19:19:11.814395 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:11.814352    4511 generic.go:356] "Generic (PLEG): container finished" podID="30879961-3956-4ffb-b399-7c406c65e0c8" containerID="4a296a81b2886288352aa29970547993a833056f5e8d745af4ebcd351ecbd0c9" exitCode=0
Oct 30 19:19:11.814395 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:11.814384    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5" event={"ID":"30879961-3956-4ffb-b399-7c406c65e0c8","Type":"ContainerDied","Data":"4a296a81b2886288352aa29970547993a833056f5e8d745af4ebcd351ecbd0c9"}
Oct 30 19:19:12.507614 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.507585    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:19:12.623589 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.623520    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30879961-3956-4ffb-b399-7c406c65e0c8-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30879961-3956-4ffb-b399-7c406c65e0c8-cabundle-cert\") pod \"30879961-3956-4ffb-b399-7c406c65e0c8\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") "
Oct 30 19:19:12.623589 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.623570    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30879961-3956-4ffb-b399-7c406c65e0c8-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30879961-3956-4ffb-b399-7c406c65e0c8-kserve-provision-location\") pod \"30879961-3956-4ffb-b399-7c406c65e0c8\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") "
Oct 30 19:19:12.623797 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.623614    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30879961-3956-4ffb-b399-7c406c65e0c8-kube-api-access-d2xm5\" (UniqueName: \"kubernetes.io/projected/30879961-3956-4ffb-b399-7c406c65e0c8-kube-api-access-d2xm5\") pod \"30879961-3956-4ffb-b399-7c406c65e0c8\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") "
Oct 30 19:19:12.623797 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.623639    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30879961-3956-4ffb-b399-7c406c65e0c8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30879961-3956-4ffb-b399-7c406c65e0c8-kube-rbac-proxy-sar-config\") pod \"30879961-3956-4ffb-b399-7c406c65e0c8\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") "
Oct 30 19:19:12.623797 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.623675    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30879961-3956-4ffb-b399-7c406c65e0c8-proxy-tls\" (UniqueName: \"kubernetes.io/secret/30879961-3956-4ffb-b399-7c406c65e0c8-proxy-tls\") pod \"30879961-3956-4ffb-b399-7c406c65e0c8\" (UID: \"30879961-3956-4ffb-b399-7c406c65e0c8\") "
Oct 30 19:19:12.623997 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.623967    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30879961-3956-4ffb-b399-7c406c65e0c8-cabundle-cert" pod "30879961-3956-4ffb-b399-7c406c65e0c8" (UID: "30879961-3956-4ffb-b399-7c406c65e0c8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:12.624051 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.623996    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30879961-3956-4ffb-b399-7c406c65e0c8-kube-rbac-proxy-sar-config" pod "30879961-3956-4ffb-b399-7c406c65e0c8" (UID: "30879961-3956-4ffb-b399-7c406c65e0c8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:12.624512 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.624494    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30879961-3956-4ffb-b399-7c406c65e0c8-kserve-provision-location" pod "30879961-3956-4ffb-b399-7c406c65e0c8" (UID: "30879961-3956-4ffb-b399-7c406c65e0c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:12.625876 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.625856    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30879961-3956-4ffb-b399-7c406c65e0c8-kube-api-access-d2xm5" pod "30879961-3956-4ffb-b399-7c406c65e0c8" (UID: "30879961-3956-4ffb-b399-7c406c65e0c8"). InnerVolumeSpecName "kube-api-access-d2xm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:12.625876 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.625862    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30879961-3956-4ffb-b399-7c406c65e0c8-proxy-tls" pod "30879961-3956-4ffb-b399-7c406c65e0c8" (UID: "30879961-3956-4ffb-b399-7c406c65e0c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:12.724707 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.724675    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30879961-3956-4ffb-b399-7c406c65e0c8-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:12.724707 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.724703    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30879961-3956-4ffb-b399-7c406c65e0c8-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:12.724917 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.724719    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2xm5\" (UniqueName: \"kubernetes.io/projected/30879961-3956-4ffb-b399-7c406c65e0c8-kube-api-access-d2xm5\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:12.724917 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.724733    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30879961-3956-4ffb-b399-7c406c65e0c8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:12.724917 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.724747    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30879961-3956-4ffb-b399-7c406c65e0c8-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:12.820647 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.820612    4511 generic.go:356] "Generic (PLEG): container finished" podID="30879961-3956-4ffb-b399-7c406c65e0c8" containerID="64f985d1aaf92eddb3f5ddaf4651fb271957644a6cc9f4971c0421244e9fc04f" exitCode=0
Oct 30 19:19:12.820773 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.820669    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"
Oct 30 19:19:12.820773 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.820682    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5" event={"ID":"30879961-3956-4ffb-b399-7c406c65e0c8","Type":"ContainerDied","Data":"64f985d1aaf92eddb3f5ddaf4651fb271957644a6cc9f4971c0421244e9fc04f"}
Oct 30 19:19:12.820773 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.820728    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5" event={"ID":"30879961-3956-4ffb-b399-7c406c65e0c8","Type":"ContainerDied","Data":"5abb3d480c0e6d64aa3414f1198f3646a9067818e5c479d56a4f64c8646401d8"}
Oct 30 19:19:12.820773 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.820760    4511 scope.go:122] "RemoveContainer" containerID="4a296a81b2886288352aa29970547993a833056f5e8d745af4ebcd351ecbd0c9"
Oct 30 19:19:12.829562 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.829543    4511 scope.go:122] "RemoveContainer" containerID="64f985d1aaf92eddb3f5ddaf4651fb271957644a6cc9f4971c0421244e9fc04f"
Oct 30 19:19:12.836694 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.836679    4511 scope.go:122] "RemoveContainer" containerID="faf7d714c7407d4747f4a514d179bef05d99a12c0fe9fc648f8a44663b574a3a"
Oct 30 19:19:12.843546 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.843530    4511 scope.go:122] "RemoveContainer" containerID="4a296a81b2886288352aa29970547993a833056f5e8d745af4ebcd351ecbd0c9"
Oct 30 19:19:12.843825 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:19:12.843799    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a296a81b2886288352aa29970547993a833056f5e8d745af4ebcd351ecbd0c9\": container with ID starting with 4a296a81b2886288352aa29970547993a833056f5e8d745af4ebcd351ecbd0c9 not found: ID does not exist" containerID="4a296a81b2886288352aa29970547993a833056f5e8d745af4ebcd351ecbd0c9"
Oct 30 19:19:12.843881 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.843837    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a296a81b2886288352aa29970547993a833056f5e8d745af4ebcd351ecbd0c9"} err="failed to get container status \"4a296a81b2886288352aa29970547993a833056f5e8d745af4ebcd351ecbd0c9\": rpc error: code = NotFound desc = could not find container \"4a296a81b2886288352aa29970547993a833056f5e8d745af4ebcd351ecbd0c9\": container with ID starting with 4a296a81b2886288352aa29970547993a833056f5e8d745af4ebcd351ecbd0c9 not found: ID does not exist"
Oct 30 19:19:12.843881 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.843855    4511 scope.go:122] "RemoveContainer" containerID="64f985d1aaf92eddb3f5ddaf4651fb271957644a6cc9f4971c0421244e9fc04f"
Oct 30 19:19:12.844098 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:19:12.844080    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f985d1aaf92eddb3f5ddaf4651fb271957644a6cc9f4971c0421244e9fc04f\": container with ID starting with 64f985d1aaf92eddb3f5ddaf4651fb271957644a6cc9f4971c0421244e9fc04f not found: ID does not exist" containerID="64f985d1aaf92eddb3f5ddaf4651fb271957644a6cc9f4971c0421244e9fc04f"
Oct 30 19:19:12.844173 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.844106    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f985d1aaf92eddb3f5ddaf4651fb271957644a6cc9f4971c0421244e9fc04f"} err="failed to get container status \"64f985d1aaf92eddb3f5ddaf4651fb271957644a6cc9f4971c0421244e9fc04f\": rpc error: code = NotFound desc = could not find container \"64f985d1aaf92eddb3f5ddaf4651fb271957644a6cc9f4971c0421244e9fc04f\": container with ID starting with 64f985d1aaf92eddb3f5ddaf4651fb271957644a6cc9f4971c0421244e9fc04f not found: ID does not exist"
Oct 30 19:19:12.844173 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.844147    4511 scope.go:122] "RemoveContainer" containerID="faf7d714c7407d4747f4a514d179bef05d99a12c0fe9fc648f8a44663b574a3a"
Oct 30 19:19:12.844404 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:19:12.844388    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faf7d714c7407d4747f4a514d179bef05d99a12c0fe9fc648f8a44663b574a3a\": container with ID starting with faf7d714c7407d4747f4a514d179bef05d99a12c0fe9fc648f8a44663b574a3a not found: ID does not exist" containerID="faf7d714c7407d4747f4a514d179bef05d99a12c0fe9fc648f8a44663b574a3a"
Oct 30 19:19:12.844444 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.844410    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faf7d714c7407d4747f4a514d179bef05d99a12c0fe9fc648f8a44663b574a3a"} err="failed to get container status \"faf7d714c7407d4747f4a514d179bef05d99a12c0fe9fc648f8a44663b574a3a\": rpc error: code = NotFound desc = could not find container \"faf7d714c7407d4747f4a514d179bef05d99a12c0fe9fc648f8a44663b574a3a\": container with ID starting with faf7d714c7407d4747f4a514d179bef05d99a12c0fe9fc648f8a44663b574a3a not found: ID does not exist"
Oct 30 19:19:12.980017 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.979956    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"]
Oct 30 19:19:12.986645 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:12.986616    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-autogluon-ts-v1-noslash-predictor-69d7db45b5-pfpz5"]
Oct 30 19:19:13.655547 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:13.655514    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30879961-3956-4ffb-b399-7c406c65e0c8" path="/var/lib/kubelet/pods/30879961-3956-4ffb-b399-7c406c65e0c8/volumes"
Oct 30 19:19:17.216848 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.216815    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"]
Oct 30 19:19:17.217250 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.217114    4511 cpu_manager.go:414] "containerMap: removing container" podUID="30879961-3956-4ffb-b399-7c406c65e0c8" containerName="kube-rbac-proxy"
Oct 30 19:19:17.217250 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.217143    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30879961-3956-4ffb-b399-7c406c65e0c8" containerName="kube-rbac-proxy"
Oct 30 19:19:17.217250 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.217155    4511 cpu_manager.go:414] "containerMap: removing container" podUID="30879961-3956-4ffb-b399-7c406c65e0c8" containerName="kserve-container"
Oct 30 19:19:17.217250 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.217161    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30879961-3956-4ffb-b399-7c406c65e0c8" containerName="kserve-container"
Oct 30 19:19:17.217250 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.217179    4511 cpu_manager.go:414] "containerMap: removing container" podUID="30879961-3956-4ffb-b399-7c406c65e0c8" containerName="storage-initializer"
Oct 30 19:19:17.217250 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.217187    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30879961-3956-4ffb-b399-7c406c65e0c8" containerName="storage-initializer"
Oct 30 19:19:17.217250 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.217237    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="30879961-3956-4ffb-b399-7c406c65e0c8" containerName="kube-rbac-proxy"
Oct 30 19:19:17.217250 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.217245    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="30879961-3956-4ffb-b399-7c406c65e0c8" containerName="kserve-container"
Oct 30 19:19:17.221900 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.221880    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:17.255367 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.255344    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bdc5f70-97fe-41a6-8010-57b8a6b99578-proxy-tls\") pod \"isvc-canary-create-predictor-657958dc5c-zvzwj\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:17.255473 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.255377    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-657958dc5c-zvzwj\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:17.255473 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.255400    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kserve-provision-location\") pod \"isvc-canary-create-predictor-657958dc5c-zvzwj\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:17.255556 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.255522    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bdc5f70-97fe-41a6-8010-57b8a6b99578-cabundle-cert\") pod \"isvc-canary-create-predictor-657958dc5c-zvzwj\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:17.255641 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.255566    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl6vg\" (UniqueName: \"kubernetes.io/projected/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kube-api-access-bl6vg\") pod \"isvc-canary-create-predictor-657958dc5c-zvzwj\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:17.273267 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.273242    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Oct 30 19:19:17.320513 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.320486    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:19:17.320513 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.320502    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:19:17.320513 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.320509    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:19:17.320691 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.320507    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-create-predictor-serving-cert\""
Oct 30 19:19:17.320691 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.320483    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:19:17.320691 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.320594    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:19:17.321845 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.321829    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"]
Oct 30 19:19:17.356450 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.356425    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bdc5f70-97fe-41a6-8010-57b8a6b99578-cabundle-cert\") pod \"isvc-canary-create-predictor-657958dc5c-zvzwj\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:17.356571 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.356464    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl6vg\" (UniqueName: \"kubernetes.io/projected/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kube-api-access-bl6vg\") pod \"isvc-canary-create-predictor-657958dc5c-zvzwj\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:17.356571 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.356501    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bdc5f70-97fe-41a6-8010-57b8a6b99578-proxy-tls\") pod \"isvc-canary-create-predictor-657958dc5c-zvzwj\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:17.356571 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.356519    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-657958dc5c-zvzwj\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:17.356709 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.356641    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kserve-provision-location\") pod \"isvc-canary-create-predictor-657958dc5c-zvzwj\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:17.357002 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.356982    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kserve-provision-location\") pod \"isvc-canary-create-predictor-657958dc5c-zvzwj\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:17.357259 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.357215    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bdc5f70-97fe-41a6-8010-57b8a6b99578-cabundle-cert\") pod \"isvc-canary-create-predictor-657958dc5c-zvzwj\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:17.357259 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.357235    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-657958dc5c-zvzwj\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:17.358999 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.358981    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bdc5f70-97fe-41a6-8010-57b8a6b99578-proxy-tls\") pod \"isvc-canary-create-predictor-657958dc5c-zvzwj\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:17.416448 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.416421    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl6vg\" (UniqueName: \"kubernetes.io/projected/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kube-api-access-bl6vg\") pod \"isvc-canary-create-predictor-657958dc5c-zvzwj\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") " pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:17.538540 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.538477    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:17.959363 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:17.959326    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"]
Oct 30 19:19:17.965749 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:19:17.965726    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bdc5f70_97fe_41a6_8010_57b8a6b99578.slice/crio-57e91bfb09880eac269f880f17e3f6f476c85a7377b1dfa6b6cfe65307d0def2 WatchSource:0}: Error finding container 57e91bfb09880eac269f880f17e3f6f476c85a7377b1dfa6b6cfe65307d0def2: Status 404 returned error can't find the container with id 57e91bfb09880eac269f880f17e3f6f476c85a7377b1dfa6b6cfe65307d0def2
Oct 30 19:19:18.846391 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:18.846353    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" event={"ID":"5bdc5f70-97fe-41a6-8010-57b8a6b99578","Type":"ContainerStarted","Data":"c10575d499e8388ef5ca04e3279db9c3a05f3576440807a16ba0d16f52fcfaf8"}
Oct 30 19:19:18.846783 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:18.846407    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" event={"ID":"5bdc5f70-97fe-41a6-8010-57b8a6b99578","Type":"ContainerStarted","Data":"57e91bfb09880eac269f880f17e3f6f476c85a7377b1dfa6b6cfe65307d0def2"}
Oct 30 19:19:21.859736 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:21.859687    4511 generic.go:356] "Generic (PLEG): container finished" podID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerID="c10575d499e8388ef5ca04e3279db9c3a05f3576440807a16ba0d16f52fcfaf8" exitCode=0
Oct 30 19:19:21.859736 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:21.859723    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" event={"ID":"5bdc5f70-97fe-41a6-8010-57b8a6b99578","Type":"ContainerDied","Data":"c10575d499e8388ef5ca04e3279db9c3a05f3576440807a16ba0d16f52fcfaf8"}
Oct 30 19:19:21.860998 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:21.860982    4511 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:19:36.935645 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:36.935609    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" event={"ID":"5bdc5f70-97fe-41a6-8010-57b8a6b99578","Type":"ContainerStarted","Data":"f7ca4318e1bda44f42842f63d467cab4216ac5857b92f48216c9a8db4e9dd232"}
Oct 30 19:19:36.936037 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:36.935695    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" event={"ID":"5bdc5f70-97fe-41a6-8010-57b8a6b99578","Type":"ContainerStarted","Data":"646d0759caba5fd195d0f3affbfdc3d1a138a8bdb86c27108d44913d0efb9a14"}
Oct 30 19:19:36.936037 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:36.935946    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:37.098768 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:37.098713    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podStartSLOduration=1.90775553 podStartE2EDuration="20.098700279s" totalImagesPullingTime="14.464823259s" totalInitContainerRuntime="3.72612149s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:37.098386128 +0000 UTC m=+2471.616071031" watchObservedRunningTime="2026-10-30 19:19:37.098700279 +0000 UTC m=+2471.616385176"
Oct 30 19:19:37.939289 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:37.939263    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:37.940519 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:37.940491    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Oct 30 19:19:38.943320 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:38.943284    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Oct 30 19:19:40.155812 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:19:40.155781    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:19:43.948037 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:43.948010    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:19:43.948489 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:43.948464    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Oct 30 19:19:53.948831 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:53.948792    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Oct 30 19:19:55.405778 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.405745    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"]
Oct 30 19:19:55.630979 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.630944    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"]
Oct 30 19:19:55.631154 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.631066    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:19:55.681670 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.681617    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:19:55.681786 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.681750    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 30 19:19:55.681855 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.681838    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:19:55.729421 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.729396    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:19:55.729421 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.729403    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:19:55.729570 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.729424    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Oct 30 19:19:55.729570 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.729490    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:19:55.759311 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.759282    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1e74e4b-cb86-4274-8879-0a3859f6e855-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:19:55.759439 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.759341    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1e74e4b-cb86-4274-8879-0a3859f6e855-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:19:55.759505 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.759435    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1e74e4b-cb86-4274-8879-0a3859f6e855-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:19:55.759505 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.759488    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1e74e4b-cb86-4274-8879-0a3859f6e855-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:19:55.759591 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.759516    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnhx9\" (UniqueName: \"kubernetes.io/projected/c1e74e4b-cb86-4274-8879-0a3859f6e855-kube-api-access-lnhx9\") pod \"isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:19:55.860707 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.860677    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1e74e4b-cb86-4274-8879-0a3859f6e855-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:19:55.860863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.860736    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1e74e4b-cb86-4274-8879-0a3859f6e855-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:19:55.860863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.860760    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1e74e4b-cb86-4274-8879-0a3859f6e855-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:19:55.860863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.860786    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1e74e4b-cb86-4274-8879-0a3859f6e855-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:19:55.860863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.860816    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnhx9\" (UniqueName: \"kubernetes.io/projected/c1e74e4b-cb86-4274-8879-0a3859f6e855-kube-api-access-lnhx9\") pod \"isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:19:55.861298 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.861277    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1e74e4b-cb86-4274-8879-0a3859f6e855-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:19:55.861542 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.861497    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1e74e4b-cb86-4274-8879-0a3859f6e855-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:19:55.861542 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.861545    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1e74e4b-cb86-4274-8879-0a3859f6e855-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:19:55.863419 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.863398    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1e74e4b-cb86-4274-8879-0a3859f6e855-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:19:55.920571 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.920538    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnhx9\" (UniqueName: \"kubernetes.io/projected/c1e74e4b-cb86-4274-8879-0a3859f6e855-kube-api-access-lnhx9\") pod \"isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:19:55.992739 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:55.992674    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:19:56.428584 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:56.428551    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"]
Oct 30 19:19:56.435471 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:19:56.435448    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e74e4b_cb86_4274_8879_0a3859f6e855.slice/crio-00ad964d5ab28a0b5d9e46eccdb629d9f1ad617b061e73d88105beb9408b8538 WatchSource:0}: Error finding container 00ad964d5ab28a0b5d9e46eccdb629d9f1ad617b061e73d88105beb9408b8538: Status 404 returned error can't find the container with id 00ad964d5ab28a0b5d9e46eccdb629d9f1ad617b061e73d88105beb9408b8538
Oct 30 19:19:57.008004 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:57.007968    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv" event={"ID":"c1e74e4b-cb86-4274-8879-0a3859f6e855","Type":"ContainerStarted","Data":"43d8897a4ead7076891f1805dfe1a97cb425b575125adbe53051e2be96413d88"}
Oct 30 19:19:57.008202 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:19:57.008037    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv" event={"ID":"c1e74e4b-cb86-4274-8879-0a3859f6e855","Type":"ContainerStarted","Data":"00ad964d5ab28a0b5d9e46eccdb629d9f1ad617b061e73d88105beb9408b8538"}
Oct 30 19:20:01.026355 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:01.026322    4511 generic.go:356] "Generic (PLEG): container finished" podID="c1e74e4b-cb86-4274-8879-0a3859f6e855" containerID="43d8897a4ead7076891f1805dfe1a97cb425b575125adbe53051e2be96413d88" exitCode=0
Oct 30 19:20:01.026779 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:01.026407    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv" event={"ID":"c1e74e4b-cb86-4274-8879-0a3859f6e855","Type":"ContainerDied","Data":"43d8897a4ead7076891f1805dfe1a97cb425b575125adbe53051e2be96413d88"}
Oct 30 19:20:02.032047 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:02.032013    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv" event={"ID":"c1e74e4b-cb86-4274-8879-0a3859f6e855","Type":"ContainerStarted","Data":"b06943a39d595e3718c93cbc33556cf4987123ccfc3a227d04040d303165c772"}
Oct 30 19:20:02.032482 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:02.032078    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv" event={"ID":"c1e74e4b-cb86-4274-8879-0a3859f6e855","Type":"ContainerStarted","Data":"e03cecf833e0fa98babb1c31bd245753508446b4efaf6baef10effead22d8229"}
Oct 30 19:20:02.032482 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:02.032384    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:20:02.032482 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:02.032435    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:20:02.148299 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:02.148238    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv" podStartSLOduration=3.020395688 podStartE2EDuration="7.148224866s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.127829178s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:02.148105257 +0000 UTC m=+2496.665790157" watchObservedRunningTime="2026-10-30 19:20:02.148224866 +0000 UTC m=+2496.665909803"
Oct 30 19:20:03.949064 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:03.949027    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Oct 30 19:20:08.041454 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:08.041424    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:20:10.156264 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:20:10.156229    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:20:13.948746 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:13.948701    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Oct 30 19:20:23.949235 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:23.949193    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Oct 30 19:20:33.949428 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:33.949385    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Oct 30 19:20:38.045418 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:38.045385    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:20:40.156848 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:20:40.156810    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:20:41.106827 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:41.106797    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"]
Oct 30 19:20:41.107093 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:41.107049    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kserve-container" containerID="cri-o://646d0759caba5fd195d0f3affbfdc3d1a138a8bdb86c27108d44913d0efb9a14" gracePeriod=30
Oct 30 19:20:41.107172 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:41.107099    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kube-rbac-proxy" containerID="cri-o://f7ca4318e1bda44f42842f63d467cab4216ac5857b92f48216c9a8db4e9dd232" gracePeriod=30
Oct 30 19:20:41.107306 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:41.107280    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Oct 30 19:20:42.191112 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:42.191080    4511 generic.go:356] "Generic (PLEG): container finished" podID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerID="f7ca4318e1bda44f42842f63d467cab4216ac5857b92f48216c9a8db4e9dd232" exitCode=0
Oct 30 19:20:42.191505 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:42.191159    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" event={"ID":"5bdc5f70-97fe-41a6-8010-57b8a6b99578","Type":"ContainerDied","Data":"f7ca4318e1bda44f42842f63d467cab4216ac5857b92f48216c9a8db4e9dd232"}
Oct 30 19:20:43.944190 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:43.944154    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Oct 30 19:20:45.517244 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:45.517211    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"]
Oct 30 19:20:45.517627 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:45.517453    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv" podUID="c1e74e4b-cb86-4274-8879-0a3859f6e855" containerName="kserve-container" containerID="cri-o://e03cecf833e0fa98babb1c31bd245753508446b4efaf6baef10effead22d8229" gracePeriod=30
Oct 30 19:20:45.517627 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:45.517498    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv" podUID="c1e74e4b-cb86-4274-8879-0a3859f6e855" containerName="kube-rbac-proxy" containerID="cri-o://b06943a39d595e3718c93cbc33556cf4987123ccfc3a227d04040d303165c772" gracePeriod=30
Oct 30 19:20:46.207904 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:46.207874    4511 generic.go:356] "Generic (PLEG): container finished" podID="c1e74e4b-cb86-4274-8879-0a3859f6e855" containerID="b06943a39d595e3718c93cbc33556cf4987123ccfc3a227d04040d303165c772" exitCode=0
Oct 30 19:20:46.208089 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:46.207946    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv" event={"ID":"c1e74e4b-cb86-4274-8879-0a3859f6e855","Type":"ContainerDied","Data":"b06943a39d595e3718c93cbc33556cf4987123ccfc3a227d04040d303165c772"}
Oct 30 19:20:48.036670 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:48.036624    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv" podUID="c1e74e4b-cb86-4274-8879-0a3859f6e855" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Oct 30 19:20:48.943586 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:48.943546    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Oct 30 19:20:49.083271 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:49.083228    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv" podUID="c1e74e4b-cb86-4274-8879-0a3859f6e855" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.45:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:20:51.108827 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:51.108798    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:20:53.036053 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:53.036005    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv" podUID="c1e74e4b-cb86-4274-8879-0a3859f6e855" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Oct 30 19:20:53.944457 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:53.944419    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Oct 30 19:20:53.944633 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:53.944553    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:20:54.759846 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:54.759824    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:20:54.911328 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:54.911259    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1e74e4b-cb86-4274-8879-0a3859f6e855-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1e74e4b-cb86-4274-8879-0a3859f6e855-proxy-tls\") pod \"c1e74e4b-cb86-4274-8879-0a3859f6e855\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") "
Oct 30 19:20:54.911457 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:54.911329    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e74e4b-cb86-4274-8879-0a3859f6e855-kube-api-access-lnhx9\" (UniqueName: \"kubernetes.io/projected/c1e74e4b-cb86-4274-8879-0a3859f6e855-kube-api-access-lnhx9\") pod \"c1e74e4b-cb86-4274-8879-0a3859f6e855\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") "
Oct 30 19:20:54.911457 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:54.911364    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e74e4b-cb86-4274-8879-0a3859f6e855-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1e74e4b-cb86-4274-8879-0a3859f6e855-kserve-provision-location\") pod \"c1e74e4b-cb86-4274-8879-0a3859f6e855\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") "
Oct 30 19:20:54.911457 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:54.911391    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1e74e4b-cb86-4274-8879-0a3859f6e855-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1e74e4b-cb86-4274-8879-0a3859f6e855-kube-rbac-proxy-sar-config\") pod \"c1e74e4b-cb86-4274-8879-0a3859f6e855\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") "
Oct 30 19:20:54.911457 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:54.911412    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1e74e4b-cb86-4274-8879-0a3859f6e855-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1e74e4b-cb86-4274-8879-0a3859f6e855-cabundle-cert\") pod \"c1e74e4b-cb86-4274-8879-0a3859f6e855\" (UID: \"c1e74e4b-cb86-4274-8879-0a3859f6e855\") "
Oct 30 19:20:54.911761 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:54.911728    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e74e4b-cb86-4274-8879-0a3859f6e855-kserve-provision-location" pod "c1e74e4b-cb86-4274-8879-0a3859f6e855" (UID: "c1e74e4b-cb86-4274-8879-0a3859f6e855"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:54.911761 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:54.911736    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e74e4b-cb86-4274-8879-0a3859f6e855-kube-rbac-proxy-sar-config" pod "c1e74e4b-cb86-4274-8879-0a3859f6e855" (UID: "c1e74e4b-cb86-4274-8879-0a3859f6e855"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:54.911950 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:54.911778    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e74e4b-cb86-4274-8879-0a3859f6e855-cabundle-cert" pod "c1e74e4b-cb86-4274-8879-0a3859f6e855" (UID: "c1e74e4b-cb86-4274-8879-0a3859f6e855"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:54.913731 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:54.913710    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1e74e4b-cb86-4274-8879-0a3859f6e855-proxy-tls" pod "c1e74e4b-cb86-4274-8879-0a3859f6e855" (UID: "c1e74e4b-cb86-4274-8879-0a3859f6e855"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:54.913787 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:54.913765    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e74e4b-cb86-4274-8879-0a3859f6e855-kube-api-access-lnhx9" pod "c1e74e4b-cb86-4274-8879-0a3859f6e855" (UID: "c1e74e4b-cb86-4274-8879-0a3859f6e855"). InnerVolumeSpecName "kube-api-access-lnhx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:55.011955 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.011931    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnhx9\" (UniqueName: \"kubernetes.io/projected/c1e74e4b-cb86-4274-8879-0a3859f6e855-kube-api-access-lnhx9\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:55.011955 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.011954    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1e74e4b-cb86-4274-8879-0a3859f6e855-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:55.012112 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.011964    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1e74e4b-cb86-4274-8879-0a3859f6e855-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:55.012112 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.011974    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1e74e4b-cb86-4274-8879-0a3859f6e855-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:55.012112 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.011983    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1e74e4b-cb86-4274-8879-0a3859f6e855-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:55.243431 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.243354    4511 generic.go:356] "Generic (PLEG): container finished" podID="c1e74e4b-cb86-4274-8879-0a3859f6e855" containerID="e03cecf833e0fa98babb1c31bd245753508446b4efaf6baef10effead22d8229" exitCode=0
Oct 30 19:20:55.243431 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.243420    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv" event={"ID":"c1e74e4b-cb86-4274-8879-0a3859f6e855","Type":"ContainerDied","Data":"e03cecf833e0fa98babb1c31bd245753508446b4efaf6baef10effead22d8229"}
Oct 30 19:20:55.243596 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.243467    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"
Oct 30 19:20:55.243596 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.243479    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv" event={"ID":"c1e74e4b-cb86-4274-8879-0a3859f6e855","Type":"ContainerDied","Data":"00ad964d5ab28a0b5d9e46eccdb629d9f1ad617b061e73d88105beb9408b8538"}
Oct 30 19:20:55.243596 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.243517    4511 scope.go:122] "RemoveContainer" containerID="b06943a39d595e3718c93cbc33556cf4987123ccfc3a227d04040d303165c772"
Oct 30 19:20:55.252822 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.252805    4511 scope.go:122] "RemoveContainer" containerID="e03cecf833e0fa98babb1c31bd245753508446b4efaf6baef10effead22d8229"
Oct 30 19:20:55.260508 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.260491    4511 scope.go:122] "RemoveContainer" containerID="43d8897a4ead7076891f1805dfe1a97cb425b575125adbe53051e2be96413d88"
Oct 30 19:20:55.267576 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.267560    4511 scope.go:122] "RemoveContainer" containerID="b06943a39d595e3718c93cbc33556cf4987123ccfc3a227d04040d303165c772"
Oct 30 19:20:55.267845 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:20:55.267826    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06943a39d595e3718c93cbc33556cf4987123ccfc3a227d04040d303165c772\": container with ID starting with b06943a39d595e3718c93cbc33556cf4987123ccfc3a227d04040d303165c772 not found: ID does not exist" containerID="b06943a39d595e3718c93cbc33556cf4987123ccfc3a227d04040d303165c772"
Oct 30 19:20:55.267910 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.267853    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06943a39d595e3718c93cbc33556cf4987123ccfc3a227d04040d303165c772"} err="failed to get container status \"b06943a39d595e3718c93cbc33556cf4987123ccfc3a227d04040d303165c772\": rpc error: code = NotFound desc = could not find container \"b06943a39d595e3718c93cbc33556cf4987123ccfc3a227d04040d303165c772\": container with ID starting with b06943a39d595e3718c93cbc33556cf4987123ccfc3a227d04040d303165c772 not found: ID does not exist"
Oct 30 19:20:55.267910 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.267871    4511 scope.go:122] "RemoveContainer" containerID="e03cecf833e0fa98babb1c31bd245753508446b4efaf6baef10effead22d8229"
Oct 30 19:20:55.268149 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:20:55.268110    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e03cecf833e0fa98babb1c31bd245753508446b4efaf6baef10effead22d8229\": container with ID starting with e03cecf833e0fa98babb1c31bd245753508446b4efaf6baef10effead22d8229 not found: ID does not exist" containerID="e03cecf833e0fa98babb1c31bd245753508446b4efaf6baef10effead22d8229"
Oct 30 19:20:55.268209 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.268158    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e03cecf833e0fa98babb1c31bd245753508446b4efaf6baef10effead22d8229"} err="failed to get container status \"e03cecf833e0fa98babb1c31bd245753508446b4efaf6baef10effead22d8229\": rpc error: code = NotFound desc = could not find container \"e03cecf833e0fa98babb1c31bd245753508446b4efaf6baef10effead22d8229\": container with ID starting with e03cecf833e0fa98babb1c31bd245753508446b4efaf6baef10effead22d8229 not found: ID does not exist"
Oct 30 19:20:55.268209 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.268177    4511 scope.go:122] "RemoveContainer" containerID="43d8897a4ead7076891f1805dfe1a97cb425b575125adbe53051e2be96413d88"
Oct 30 19:20:55.268421 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:20:55.268401    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d8897a4ead7076891f1805dfe1a97cb425b575125adbe53051e2be96413d88\": container with ID starting with 43d8897a4ead7076891f1805dfe1a97cb425b575125adbe53051e2be96413d88 not found: ID does not exist" containerID="43d8897a4ead7076891f1805dfe1a97cb425b575125adbe53051e2be96413d88"
Oct 30 19:20:55.268481 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.268425    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d8897a4ead7076891f1805dfe1a97cb425b575125adbe53051e2be96413d88"} err="failed to get container status \"43d8897a4ead7076891f1805dfe1a97cb425b575125adbe53051e2be96413d88\": rpc error: code = NotFound desc = could not find container \"43d8897a4ead7076891f1805dfe1a97cb425b575125adbe53051e2be96413d88\": container with ID starting with 43d8897a4ead7076891f1805dfe1a97cb425b575125adbe53051e2be96413d88 not found: ID does not exist"
Oct 30 19:20:55.401762 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.401741    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"]
Oct 30 19:20:55.449179 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.449158    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-mlserver-predictor-654bdcc6b6-hzczv"]
Oct 30 19:20:55.656342 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:55.656314    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1e74e4b-cb86-4274-8879-0a3859f6e855" path="/var/lib/kubelet/pods/c1e74e4b-cb86-4274-8879-0a3859f6e855/volumes"
Oct 30 19:20:58.727602 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.727567    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"]
Oct 30 19:20:58.727969 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.727859    4511 cpu_manager.go:414] "containerMap: removing container" podUID="c1e74e4b-cb86-4274-8879-0a3859f6e855" containerName="kube-rbac-proxy"
Oct 30 19:20:58.727969 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.727870    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e74e4b-cb86-4274-8879-0a3859f6e855" containerName="kube-rbac-proxy"
Oct 30 19:20:58.727969 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.727882    4511 cpu_manager.go:414] "containerMap: removing container" podUID="c1e74e4b-cb86-4274-8879-0a3859f6e855" containerName="kserve-container"
Oct 30 19:20:58.727969 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.727887    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e74e4b-cb86-4274-8879-0a3859f6e855" containerName="kserve-container"
Oct 30 19:20:58.727969 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.727897    4511 cpu_manager.go:414] "containerMap: removing container" podUID="c1e74e4b-cb86-4274-8879-0a3859f6e855" containerName="storage-initializer"
Oct 30 19:20:58.727969 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.727902    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e74e4b-cb86-4274-8879-0a3859f6e855" containerName="storage-initializer"
Oct 30 19:20:58.727969 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.727953    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e74e4b-cb86-4274-8879-0a3859f6e855" containerName="kube-rbac-proxy"
Oct 30 19:20:58.727969 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.727962    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e74e4b-cb86-4274-8879-0a3859f6e855" containerName="kserve-container"
Oct 30 19:20:58.732591 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.732570    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:20:58.788917 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.788898    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:20:58.789032 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.788919    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:20:58.836347 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.836309    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Oct 30 19:20:58.836347 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.836310    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 30 19:20:58.836347 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.836345    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:20:58.836565 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.836401    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:20:58.836565 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.836402    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:20:58.837901 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.837884    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"]
Oct 30 19:20:58.838201 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.838186    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") " pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:20:58.838241 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.838213    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") " pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:20:58.838241 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.838231    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw7ln\" (UniqueName: \"kubernetes.io/projected/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kube-api-access-sw7ln\") pod \"xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") " pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:20:58.838378 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.838362    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") " pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:20:58.838414 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.838396    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") " pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:20:58.939418 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.939392    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") " pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:20:58.939533 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.939420    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw7ln\" (UniqueName: \"kubernetes.io/projected/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kube-api-access-sw7ln\") pod \"xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") " pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:20:58.939533 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.939454    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") " pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:20:58.939533 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.939492    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") " pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:20:58.939689 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.939553    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") " pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:20:58.939903 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.939883    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") " pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:20:58.940146 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.940106    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") " pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:20:58.940232 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.940110    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") " pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:20:58.942076 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.942052    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") " pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:20:58.944187 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.944161    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Oct 30 19:20:58.998373 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:58.998312    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw7ln\" (UniqueName: \"kubernetes.io/projected/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kube-api-access-sw7ln\") pod \"xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") " pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:20:59.049730 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:59.049706    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:20:59.475499 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:20:59.475472    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"]
Oct 30 19:20:59.482738 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:20:59.482708    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bcc90a9_ec21_482e_8a7c_db42c3e03e0c.slice/crio-0829cb4dcaee8b58ed5923eba44ea6c7b7da746590e50af94db2ba29e77bc8cf WatchSource:0}: Error finding container 0829cb4dcaee8b58ed5923eba44ea6c7b7da746590e50af94db2ba29e77bc8cf: Status 404 returned error can't find the container with id 0829cb4dcaee8b58ed5923eba44ea6c7b7da746590e50af94db2ba29e77bc8cf
Oct 30 19:21:00.265719 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:00.265678    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5" event={"ID":"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c","Type":"ContainerStarted","Data":"5ac868c36b44a5c33771864f0d2e832c0a40e39f07edcf36b24f28b040ef4dc1"}
Oct 30 19:21:00.266106 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:00.265764    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5" event={"ID":"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c","Type":"ContainerStarted","Data":"0829cb4dcaee8b58ed5923eba44ea6c7b7da746590e50af94db2ba29e77bc8cf"}
Oct 30 19:21:03.944091 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:03.944049    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Oct 30 19:21:04.282725 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:04.282648    4511 generic.go:356] "Generic (PLEG): container finished" podID="7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" containerID="5ac868c36b44a5c33771864f0d2e832c0a40e39f07edcf36b24f28b040ef4dc1" exitCode=0
Oct 30 19:21:04.282725 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:04.282712    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5" event={"ID":"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c","Type":"ContainerDied","Data":"5ac868c36b44a5c33771864f0d2e832c0a40e39f07edcf36b24f28b040ef4dc1"}
Oct 30 19:21:05.289285 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:05.289252    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5" event={"ID":"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c","Type":"ContainerStarted","Data":"cd80d7d27cd6fc14930fc179c9b3098d3cb4da055104ea5644cf60ac08647337"}
Oct 30 19:21:05.289810 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:05.289311    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5" event={"ID":"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c","Type":"ContainerStarted","Data":"d91ef33a0f3ee2a4f0fe06ca0fb364ca69603e07a6118adb1aa319d1dd9a7c50"}
Oct 30 19:21:05.289810 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:05.289524    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:21:05.289810 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:05.289690    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:21:05.405681 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:05.405624    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5" podStartSLOduration=3.426014009 podStartE2EDuration="7.405612703s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.979598694s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:05.40550782 +0000 UTC m=+2559.923192720" watchObservedRunningTime="2026-10-30 19:21:05.405612703 +0000 UTC m=+2559.923297603"
Oct 30 19:21:08.944081 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:08.944038    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Oct 30 19:21:10.156423 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:21:10.156382    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:21:11.299562 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.299536    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:21:11.315942 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.315918    4511 generic.go:356] "Generic (PLEG): container finished" podID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerID="646d0759caba5fd195d0f3affbfdc3d1a138a8bdb86c27108d44913d0efb9a14" exitCode=137
Oct 30 19:21:11.316080 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.315984    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" event={"ID":"5bdc5f70-97fe-41a6-8010-57b8a6b99578","Type":"ContainerDied","Data":"646d0759caba5fd195d0f3affbfdc3d1a138a8bdb86c27108d44913d0efb9a14"}
Oct 30 19:21:11.383357 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.383335    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:21:11.544066 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.544041    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kube-api-access-bl6vg\" (UniqueName: \"kubernetes.io/projected/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kube-api-access-bl6vg\") pod \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") "
Oct 30 19:21:11.544242 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.544088    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bdc5f70-97fe-41a6-8010-57b8a6b99578-proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bdc5f70-97fe-41a6-8010-57b8a6b99578-proxy-tls\") pod \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") "
Oct 30 19:21:11.544242 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.544114    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kserve-provision-location\") pod \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") "
Oct 30 19:21:11.544242 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.544231    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kube-rbac-proxy-sar-config\") pod \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") "
Oct 30 19:21:11.544392 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.544273    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bdc5f70-97fe-41a6-8010-57b8a6b99578-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bdc5f70-97fe-41a6-8010-57b8a6b99578-cabundle-cert\") pod \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\" (UID: \"5bdc5f70-97fe-41a6-8010-57b8a6b99578\") "
Oct 30 19:21:11.544533 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.544504    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kserve-provision-location" pod "5bdc5f70-97fe-41a6-8010-57b8a6b99578" (UID: "5bdc5f70-97fe-41a6-8010-57b8a6b99578"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:11.544642 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.544587    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kube-rbac-proxy-sar-config" pod "5bdc5f70-97fe-41a6-8010-57b8a6b99578" (UID: "5bdc5f70-97fe-41a6-8010-57b8a6b99578"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:11.544713 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.544688    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bdc5f70-97fe-41a6-8010-57b8a6b99578-cabundle-cert" pod "5bdc5f70-97fe-41a6-8010-57b8a6b99578" (UID: "5bdc5f70-97fe-41a6-8010-57b8a6b99578"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:11.546323 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.546298    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bdc5f70-97fe-41a6-8010-57b8a6b99578-proxy-tls" pod "5bdc5f70-97fe-41a6-8010-57b8a6b99578" (UID: "5bdc5f70-97fe-41a6-8010-57b8a6b99578"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:11.546419 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.546361    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kube-api-access-bl6vg" pod "5bdc5f70-97fe-41a6-8010-57b8a6b99578" (UID: "5bdc5f70-97fe-41a6-8010-57b8a6b99578"). InnerVolumeSpecName "kube-api-access-bl6vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:11.644977 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.644912    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl6vg\" (UniqueName: \"kubernetes.io/projected/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kube-api-access-bl6vg\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:11.644977 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.644935    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bdc5f70-97fe-41a6-8010-57b8a6b99578-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:11.644977 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.644944    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:11.644977 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.644954    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bdc5f70-97fe-41a6-8010-57b8a6b99578-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:11.644977 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:11.644964    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bdc5f70-97fe-41a6-8010-57b8a6b99578-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:12.322048 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:12.322021    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"
Oct 30 19:21:12.322048 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:12.322031    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj" event={"ID":"5bdc5f70-97fe-41a6-8010-57b8a6b99578","Type":"ContainerDied","Data":"57e91bfb09880eac269f880f17e3f6f476c85a7377b1dfa6b6cfe65307d0def2"}
Oct 30 19:21:12.322672 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:12.322091    4511 scope.go:122] "RemoveContainer" containerID="f7ca4318e1bda44f42842f63d467cab4216ac5857b92f48216c9a8db4e9dd232"
Oct 30 19:21:12.330036 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:12.330016    4511 scope.go:122] "RemoveContainer" containerID="646d0759caba5fd195d0f3affbfdc3d1a138a8bdb86c27108d44913d0efb9a14"
Oct 30 19:21:12.337375 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:12.337359    4511 scope.go:122] "RemoveContainer" containerID="c10575d499e8388ef5ca04e3279db9c3a05f3576440807a16ba0d16f52fcfaf8"
Oct 30 19:21:12.528377 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:12.528352    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"]
Oct 30 19:21:12.576046 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:12.575971    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-create-predictor-657958dc5c-zvzwj"]
Oct 30 19:21:13.656636 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:13.656605    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" path="/var/lib/kubelet/pods/5bdc5f70-97fe-41a6-8010-57b8a6b99578/volumes"
Oct 30 19:21:14.812663 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.812628    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"]
Oct 30 19:21:14.813066 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.813052    4511 cpu_manager.go:414] "containerMap: removing container" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kserve-container"
Oct 30 19:21:14.813110 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.813069    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kserve-container"
Oct 30 19:21:14.813110 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.813098    4511 cpu_manager.go:414] "containerMap: removing container" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kube-rbac-proxy"
Oct 30 19:21:14.813110 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.813107    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kube-rbac-proxy"
Oct 30 19:21:14.813222 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.813147    4511 cpu_manager.go:414] "containerMap: removing container" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="storage-initializer"
Oct 30 19:21:14.813222 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.813156    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="storage-initializer"
Oct 30 19:21:14.813280 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.813241    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kserve-container"
Oct 30 19:21:14.813280 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.813255    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bdc5f70-97fe-41a6-8010-57b8a6b99578" containerName="kube-rbac-proxy"
Oct 30 19:21:14.817886 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.817866    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:14.868501 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.868471    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:21:14.868639 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.868621    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Oct 30 19:21:14.875189 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.875163    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-76656f7564-kwkjw\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:14.875270 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.875218    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-76656f7564-kwkjw\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:14.875309 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.875292    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n6w5\" (UniqueName: \"kubernetes.io/projected/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kube-api-access-6n6w5\") pod \"isvc-canary-promote-v2-predictor-76656f7564-kwkjw\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:14.875354 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.875324    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-76656f7564-kwkjw\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:14.875354 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.875347    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-76656f7564-kwkjw\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:14.916052 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.916022    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:21:14.916205 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.916051    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Oct 30 19:21:14.916205 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.916053    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:21:14.916205 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.916161    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:21:14.916410 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.916389    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:21:14.917355 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.917338    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"]
Oct 30 19:21:14.976221 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.976192    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-76656f7564-kwkjw\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:14.976343 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.976238    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-76656f7564-kwkjw\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:14.976343 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.976268    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-76656f7564-kwkjw\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:14.976343 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.976313    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-76656f7564-kwkjw\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:14.976343 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.976332    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n6w5\" (UniqueName: \"kubernetes.io/projected/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kube-api-access-6n6w5\") pod \"isvc-canary-promote-v2-predictor-76656f7564-kwkjw\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:14.976575 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.976556    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-76656f7564-kwkjw\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:14.977016 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.976992    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-76656f7564-kwkjw\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:14.977106 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.977004    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-76656f7564-kwkjw\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:14.978890 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:14.978873    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-76656f7564-kwkjw\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:15.036974 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.036948    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n6w5\" (UniqueName: \"kubernetes.io/projected/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kube-api-access-6n6w5\") pod \"isvc-canary-promote-v2-predictor-76656f7564-kwkjw\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:15.134327 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.134261    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:15.560161 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.560110    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"]
Oct 30 19:21:15.567256 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:21:15.567223    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f553bc_36b5_47bf_a08d_8c7e8aacd02e.slice/crio-3fa48241695b7bde4f61d244c1f354205bf1c5667c04407565ce042f7332e51a WatchSource:0}: Error finding container 3fa48241695b7bde4f61d244c1f354205bf1c5667c04407565ce042f7332e51a: Status 404 returned error can't find the container with id 3fa48241695b7bde4f61d244c1f354205bf1c5667c04407565ce042f7332e51a
Oct 30 19:21:15.800221 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.800189    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"]
Oct 30 19:21:15.803614 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.803595    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:15.854005 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.853933    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-promote-predictor-serving-cert\""
Oct 30 19:21:15.864280 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.864257    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"]
Oct 30 19:21:15.883533 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.883506    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/203f17dc-dd8b-42dc-9507-ff5afc07dd28-proxy-tls\") pod \"isvc-canary-promote-predictor-58b554b449-nnlv4\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:15.883689 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.883541    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-58b554b449-nnlv4\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:15.883689 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.883671    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/203f17dc-dd8b-42dc-9507-ff5afc07dd28-cabundle-cert\") pod \"isvc-canary-promote-predictor-58b554b449-nnlv4\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:15.883794 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.883728    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzftx\" (UniqueName: \"kubernetes.io/projected/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kube-api-access-vzftx\") pod \"isvc-canary-promote-predictor-58b554b449-nnlv4\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:15.883794 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.883758    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kserve-provision-location\") pod \"isvc-canary-promote-predictor-58b554b449-nnlv4\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:15.984224 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.984189    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-58b554b449-nnlv4\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:15.984397 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.984264    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/203f17dc-dd8b-42dc-9507-ff5afc07dd28-cabundle-cert\") pod \"isvc-canary-promote-predictor-58b554b449-nnlv4\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:15.984397 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.984296    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzftx\" (UniqueName: \"kubernetes.io/projected/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kube-api-access-vzftx\") pod \"isvc-canary-promote-predictor-58b554b449-nnlv4\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:15.984397 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.984349    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kserve-provision-location\") pod \"isvc-canary-promote-predictor-58b554b449-nnlv4\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:15.984606 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.984415    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/203f17dc-dd8b-42dc-9507-ff5afc07dd28-proxy-tls\") pod \"isvc-canary-promote-predictor-58b554b449-nnlv4\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:15.985207 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.985183    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kserve-provision-location\") pod \"isvc-canary-promote-predictor-58b554b449-nnlv4\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:15.985353 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.985332    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-58b554b449-nnlv4\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:15.985461 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.985360    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/203f17dc-dd8b-42dc-9507-ff5afc07dd28-cabundle-cert\") pod \"isvc-canary-promote-predictor-58b554b449-nnlv4\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:15.986960 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:15.986938    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/203f17dc-dd8b-42dc-9507-ff5afc07dd28-proxy-tls\") pod \"isvc-canary-promote-predictor-58b554b449-nnlv4\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:16.044087 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:16.044060    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzftx\" (UniqueName: \"kubernetes.io/projected/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kube-api-access-vzftx\") pod \"isvc-canary-promote-predictor-58b554b449-nnlv4\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") " pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:16.120295 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:16.120231    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:16.340467 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:16.340430    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" event={"ID":"77f553bc-36b5-47bf-a08d-8c7e8aacd02e","Type":"ContainerStarted","Data":"2096e70790b2b9f77ac72bb6c1bc5cdf196732caccdd28d48159589903b09420"}
Oct 30 19:21:16.340636 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:16.340497    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" event={"ID":"77f553bc-36b5-47bf-a08d-8c7e8aacd02e","Type":"ContainerStarted","Data":"3fa48241695b7bde4f61d244c1f354205bf1c5667c04407565ce042f7332e51a"}
Oct 30 19:21:16.625687 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:16.625656    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"]
Oct 30 19:21:16.631784 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:21:16.631752    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203f17dc_dd8b_42dc_9507_ff5afc07dd28.slice/crio-5b083036d77c36751f50dc8416156a56d80f852b6da9d836de7299439c484bfb WatchSource:0}: Error finding container 5b083036d77c36751f50dc8416156a56d80f852b6da9d836de7299439c484bfb: Status 404 returned error can't find the container with id 5b083036d77c36751f50dc8416156a56d80f852b6da9d836de7299439c484bfb
Oct 30 19:21:17.351091 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:17.351046    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" event={"ID":"203f17dc-dd8b-42dc-9507-ff5afc07dd28","Type":"ContainerStarted","Data":"c4d63583697f39602ad859747178f3766e712e412e0f1bc6c4d26de5243d4365"}
Oct 30 19:21:17.351550 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:17.351143    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" event={"ID":"203f17dc-dd8b-42dc-9507-ff5afc07dd28","Type":"ContainerStarted","Data":"5b083036d77c36751f50dc8416156a56d80f852b6da9d836de7299439c484bfb"}
Oct 30 19:21:20.364593 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:20.364565    4511 generic.go:356] "Generic (PLEG): container finished" podID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerID="2096e70790b2b9f77ac72bb6c1bc5cdf196732caccdd28d48159589903b09420" exitCode=0
Oct 30 19:21:20.364915 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:20.364603    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" event={"ID":"77f553bc-36b5-47bf-a08d-8c7e8aacd02e","Type":"ContainerDied","Data":"2096e70790b2b9f77ac72bb6c1bc5cdf196732caccdd28d48159589903b09420"}
Oct 30 19:21:21.370400 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:21.370365    4511 generic.go:356] "Generic (PLEG): container finished" podID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerID="c4d63583697f39602ad859747178f3766e712e412e0f1bc6c4d26de5243d4365" exitCode=0
Oct 30 19:21:21.370876 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:21.370443    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" event={"ID":"203f17dc-dd8b-42dc-9507-ff5afc07dd28","Type":"ContainerDied","Data":"c4d63583697f39602ad859747178f3766e712e412e0f1bc6c4d26de5243d4365"}
Oct 30 19:21:21.372764 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:21.372737    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" event={"ID":"77f553bc-36b5-47bf-a08d-8c7e8aacd02e","Type":"ContainerStarted","Data":"4aad789501aa370132e51c3191a7b3cee579d1b06f0bfe08947a6d425955c441"}
Oct 30 19:21:21.372852 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:21.372801    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" event={"ID":"77f553bc-36b5-47bf-a08d-8c7e8aacd02e","Type":"ContainerStarted","Data":"933ab855192afac1b36a2eb9f799c1e19f5f81c2041a83b1fa10f62e70ec904e"}
Oct 30 19:21:21.373193 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:21.373170    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:21.373280 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:21.373217    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:21.374590 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:21.374563    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Oct 30 19:21:21.602649 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:21.602595    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" podStartSLOduration=3.850079444 podStartE2EDuration="7.602578288s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.752498844s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:21.6019905 +0000 UTC m=+2576.119675415" watchObservedRunningTime="2026-10-30 19:21:21.602578288 +0000 UTC m=+2576.120263190"
Oct 30 19:21:22.379652 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:22.379621    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" event={"ID":"203f17dc-dd8b-42dc-9507-ff5afc07dd28","Type":"ContainerStarted","Data":"302cd993ef43127216d28eab9ea7acc368f3aa68379d2031a7877328dec34385"}
Oct 30 19:21:22.380034 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:22.379681    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" event={"ID":"203f17dc-dd8b-42dc-9507-ff5afc07dd28","Type":"ContainerStarted","Data":"3bfd54a5a0d2ea20ed91ef6b493556fc655ef8b04714ce845ea8c8dda26479e7"}
Oct 30 19:21:22.380034 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:22.379984    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:22.380034 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:22.380014    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Oct 30 19:21:22.496360 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:22.496311    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" podStartSLOduration=3.813826878 podStartE2EDuration="7.496297721s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.682470843s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:22.495082016 +0000 UTC m=+2577.012766917" watchObservedRunningTime="2026-10-30 19:21:22.496297721 +0000 UTC m=+2577.013982620"
Oct 30 19:21:23.383991 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:23.383961    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:23.385180 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:23.385114    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Oct 30 19:21:24.387971 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:24.387932    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Oct 30 19:21:27.385101 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:27.385073    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:21:27.385701 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:27.385671    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Oct 30 19:21:29.392175 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:29.392143    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:21:29.392636 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:29.392609    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Oct 30 19:21:37.386217 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:37.386177    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Oct 30 19:21:39.393427 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:39.393385    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Oct 30 19:21:40.155957 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:21:40.155915    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:21:41.318523 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:41.318483    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:21:47.385653 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:47.385612    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Oct 30 19:21:48.892479 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:48.892443    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"]
Oct 30 19:21:48.892871 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:48.892678    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5" podUID="7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" containerName="kserve-container" containerID="cri-o://d91ef33a0f3ee2a4f0fe06ca0fb364ca69603e07a6118adb1aa319d1dd9a7c50" gracePeriod=30
Oct 30 19:21:48.892871 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:48.892730    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5" podUID="7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" containerName="kube-rbac-proxy" containerID="cri-o://cd80d7d27cd6fc14930fc179c9b3098d3cb4da055104ea5644cf60ac08647337" gracePeriod=30
Oct 30 19:21:49.393399 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:49.393355    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Oct 30 19:21:49.496051 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:49.496019    4511 generic.go:356] "Generic (PLEG): container finished" podID="7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" containerID="cd80d7d27cd6fc14930fc179c9b3098d3cb4da055104ea5644cf60ac08647337" exitCode=0
Oct 30 19:21:49.496235 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:49.496080    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5" event={"ID":"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c","Type":"ContainerDied","Data":"cd80d7d27cd6fc14930fc179c9b3098d3cb4da055104ea5644cf60ac08647337"}
Oct 30 19:21:51.294226 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:51.294188    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5" podUID="7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Oct 30 19:21:51.300641 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:51.300614    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5" podUID="7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.46:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.133.0.46:8080: connect: connection refused"
Oct 30 19:21:54.743385 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:54.743362    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:21:54.788257 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:54.788199    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kube-rbac-proxy-sar-config\") pod \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") "
Oct 30 19:21:54.788257 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:54.788252    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-cabundle-cert\") pod \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") "
Oct 30 19:21:54.788482 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:54.788330    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-proxy-tls\") pod \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") "
Oct 30 19:21:54.788482 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:54.788381    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kserve-provision-location\") pod \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") "
Oct 30 19:21:54.788482 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:54.788418    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kube-api-access-sw7ln\" (UniqueName: \"kubernetes.io/projected/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kube-api-access-sw7ln\") pod \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\" (UID: \"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c\") "
Oct 30 19:21:54.788631 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:54.788586    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-cabundle-cert" pod "7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" (UID: "7bcc90a9-ec21-482e-8a7c-db42c3e03e0c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:54.788685 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:54.788562    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kube-rbac-proxy-sar-config" pod "7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" (UID: "7bcc90a9-ec21-482e-8a7c-db42c3e03e0c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:54.788739 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:54.788694    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kserve-provision-location" pod "7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" (UID: "7bcc90a9-ec21-482e-8a7c-db42c3e03e0c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:54.788797 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:54.788743    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:54.788797 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:54.788766    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:54.790611 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:54.790593    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-proxy-tls" pod "7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" (UID: "7bcc90a9-ec21-482e-8a7c-db42c3e03e0c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:54.790732 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:54.790713    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kube-api-access-sw7ln" pod "7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" (UID: "7bcc90a9-ec21-482e-8a7c-db42c3e03e0c"). InnerVolumeSpecName "kube-api-access-sw7ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:54.889482 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:54.889459    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:54.889482 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:54.889480    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:54.889623 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:54.889490    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw7ln\" (UniqueName: \"kubernetes.io/projected/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c-kube-api-access-sw7ln\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:55.522298 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:55.522260    4511 generic.go:356] "Generic (PLEG): container finished" podID="7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" containerID="d91ef33a0f3ee2a4f0fe06ca0fb364ca69603e07a6118adb1aa319d1dd9a7c50" exitCode=0
Oct 30 19:21:55.522515 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:55.522304    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5" event={"ID":"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c","Type":"ContainerDied","Data":"d91ef33a0f3ee2a4f0fe06ca0fb364ca69603e07a6118adb1aa319d1dd9a7c50"}
Oct 30 19:21:55.522515 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:55.522387    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5" event={"ID":"7bcc90a9-ec21-482e-8a7c-db42c3e03e0c","Type":"ContainerDied","Data":"0829cb4dcaee8b58ed5923eba44ea6c7b7da746590e50af94db2ba29e77bc8cf"}
Oct 30 19:21:55.522515 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:55.522318    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"
Oct 30 19:21:55.522515 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:55.522430    4511 scope.go:122] "RemoveContainer" containerID="cd80d7d27cd6fc14930fc179c9b3098d3cb4da055104ea5644cf60ac08647337"
Oct 30 19:21:55.531235 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:55.531217    4511 scope.go:122] "RemoveContainer" containerID="d91ef33a0f3ee2a4f0fe06ca0fb364ca69603e07a6118adb1aa319d1dd9a7c50"
Oct 30 19:21:55.538870 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:55.538855    4511 scope.go:122] "RemoveContainer" containerID="5ac868c36b44a5c33771864f0d2e832c0a40e39f07edcf36b24f28b040ef4dc1"
Oct 30 19:21:55.545826 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:55.545812    4511 scope.go:122] "RemoveContainer" containerID="cd80d7d27cd6fc14930fc179c9b3098d3cb4da055104ea5644cf60ac08647337"
Oct 30 19:21:55.546078 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:21:55.546061    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd80d7d27cd6fc14930fc179c9b3098d3cb4da055104ea5644cf60ac08647337\": container with ID starting with cd80d7d27cd6fc14930fc179c9b3098d3cb4da055104ea5644cf60ac08647337 not found: ID does not exist" containerID="cd80d7d27cd6fc14930fc179c9b3098d3cb4da055104ea5644cf60ac08647337"
Oct 30 19:21:55.546138 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:55.546087    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd80d7d27cd6fc14930fc179c9b3098d3cb4da055104ea5644cf60ac08647337"} err="failed to get container status \"cd80d7d27cd6fc14930fc179c9b3098d3cb4da055104ea5644cf60ac08647337\": rpc error: code = NotFound desc = could not find container \"cd80d7d27cd6fc14930fc179c9b3098d3cb4da055104ea5644cf60ac08647337\": container with ID starting with cd80d7d27cd6fc14930fc179c9b3098d3cb4da055104ea5644cf60ac08647337 not found: ID does not exist"
Oct 30 19:21:55.546138 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:55.546104    4511 scope.go:122] "RemoveContainer" containerID="d91ef33a0f3ee2a4f0fe06ca0fb364ca69603e07a6118adb1aa319d1dd9a7c50"
Oct 30 19:21:55.546336 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:21:55.546321    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91ef33a0f3ee2a4f0fe06ca0fb364ca69603e07a6118adb1aa319d1dd9a7c50\": container with ID starting with d91ef33a0f3ee2a4f0fe06ca0fb364ca69603e07a6118adb1aa319d1dd9a7c50 not found: ID does not exist" containerID="d91ef33a0f3ee2a4f0fe06ca0fb364ca69603e07a6118adb1aa319d1dd9a7c50"
Oct 30 19:21:55.546383 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:55.546340    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91ef33a0f3ee2a4f0fe06ca0fb364ca69603e07a6118adb1aa319d1dd9a7c50"} err="failed to get container status \"d91ef33a0f3ee2a4f0fe06ca0fb364ca69603e07a6118adb1aa319d1dd9a7c50\": rpc error: code = NotFound desc = could not find container \"d91ef33a0f3ee2a4f0fe06ca0fb364ca69603e07a6118adb1aa319d1dd9a7c50\": container with ID starting with d91ef33a0f3ee2a4f0fe06ca0fb364ca69603e07a6118adb1aa319d1dd9a7c50 not found: ID does not exist"
Oct 30 19:21:55.546383 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:55.546353    4511 scope.go:122] "RemoveContainer" containerID="5ac868c36b44a5c33771864f0d2e832c0a40e39f07edcf36b24f28b040ef4dc1"
Oct 30 19:21:55.546554 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:21:55.546539    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac868c36b44a5c33771864f0d2e832c0a40e39f07edcf36b24f28b040ef4dc1\": container with ID starting with 5ac868c36b44a5c33771864f0d2e832c0a40e39f07edcf36b24f28b040ef4dc1 not found: ID does not exist" containerID="5ac868c36b44a5c33771864f0d2e832c0a40e39f07edcf36b24f28b040ef4dc1"
Oct 30 19:21:55.546598 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:55.546557    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac868c36b44a5c33771864f0d2e832c0a40e39f07edcf36b24f28b040ef4dc1"} err="failed to get container status \"5ac868c36b44a5c33771864f0d2e832c0a40e39f07edcf36b24f28b040ef4dc1\": rpc error: code = NotFound desc = could not find container \"5ac868c36b44a5c33771864f0d2e832c0a40e39f07edcf36b24f28b040ef4dc1\": container with ID starting with 5ac868c36b44a5c33771864f0d2e832c0a40e39f07edcf36b24f28b040ef4dc1 not found: ID does not exist"
Oct 30 19:21:55.722566 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:55.722534    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"]
Oct 30 19:21:55.768751 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:55.768723    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/xgboost-v2-mlserver-predictor-7f77c6d88c-xgrq5"]
Oct 30 19:21:57.385977 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:57.385940    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Oct 30 19:21:57.655756 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:57.655682    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" path="/var/lib/kubelet/pods/7bcc90a9-ec21-482e-8a7c-db42c3e03e0c/volumes"
Oct 30 19:21:59.392836 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:21:59.392799    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Oct 30 19:22:07.385972 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:07.385932    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Oct 30 19:22:09.393407 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:09.393365    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Oct 30 19:22:10.157407 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:22:10.157366    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:22:17.385633 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:17.385592    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Oct 30 19:22:19.393074 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:19.393033    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Oct 30 19:22:27.386326 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:27.386283    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Oct 30 19:22:28.651462 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:28.651423    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Oct 30 19:22:29.392934 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:29.392899    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Oct 30 19:22:36.652394 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:36.652313    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:22:38.652290 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:38.652258    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:22:40.157558 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:22:40.157521    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:22:45.356075 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:45.356041    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"]
Oct 30 19:22:45.356616 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:45.356322    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kserve-container" containerID="cri-o://3bfd54a5a0d2ea20ed91ef6b493556fc655ef8b04714ce845ea8c8dda26479e7" gracePeriod=30
Oct 30 19:22:45.356616 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:45.356367    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kube-rbac-proxy" containerID="cri-o://302cd993ef43127216d28eab9ea7acc368f3aa68379d2031a7877328dec34385" gracePeriod=30
Oct 30 19:22:45.722090 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:45.722009    4511 generic.go:356] "Generic (PLEG): container finished" podID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerID="302cd993ef43127216d28eab9ea7acc368f3aa68379d2031a7877328dec34385" exitCode=0
Oct 30 19:22:45.722257 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:45.722080    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" event={"ID":"203f17dc-dd8b-42dc-9507-ff5afc07dd28","Type":"ContainerDied","Data":"302cd993ef43127216d28eab9ea7acc368f3aa68379d2031a7877328dec34385"}
Oct 30 19:22:46.652470 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:46.652428    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Oct 30 19:22:49.388335 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:49.388299    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Oct 30 19:22:49.896390 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:49.896365    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:22:49.997763 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:49.997701    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/203f17dc-dd8b-42dc-9507-ff5afc07dd28-proxy-tls\" (UniqueName: \"kubernetes.io/secret/203f17dc-dd8b-42dc-9507-ff5afc07dd28-proxy-tls\") pod \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") "
Oct 30 19:22:49.997763 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:49.997743    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kserve-provision-location\") pod \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") "
Oct 30 19:22:49.997955 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:49.997783    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/203f17dc-dd8b-42dc-9507-ff5afc07dd28-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/203f17dc-dd8b-42dc-9507-ff5afc07dd28-cabundle-cert\") pod \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") "
Oct 30 19:22:49.997955 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:49.997890    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kube-rbac-proxy-sar-config\") pod \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") "
Oct 30 19:22:49.997955 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:49.997941    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kube-api-access-vzftx\" (UniqueName: \"kubernetes.io/projected/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kube-api-access-vzftx\") pod \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\" (UID: \"203f17dc-dd8b-42dc-9507-ff5afc07dd28\") "
Oct 30 19:22:49.998180 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:49.998066    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kserve-provision-location" pod "203f17dc-dd8b-42dc-9507-ff5afc07dd28" (UID: "203f17dc-dd8b-42dc-9507-ff5afc07dd28"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:49.998254 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:49.998189    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/203f17dc-dd8b-42dc-9507-ff5afc07dd28-cabundle-cert" pod "203f17dc-dd8b-42dc-9507-ff5afc07dd28" (UID: "203f17dc-dd8b-42dc-9507-ff5afc07dd28"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:49.998254 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:49.998234    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kube-rbac-proxy-sar-config" pod "203f17dc-dd8b-42dc-9507-ff5afc07dd28" (UID: "203f17dc-dd8b-42dc-9507-ff5afc07dd28"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:49.999861 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:49.999840    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/203f17dc-dd8b-42dc-9507-ff5afc07dd28-proxy-tls" pod "203f17dc-dd8b-42dc-9507-ff5afc07dd28" (UID: "203f17dc-dd8b-42dc-9507-ff5afc07dd28"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:50.000070 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.000051    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kube-api-access-vzftx" pod "203f17dc-dd8b-42dc-9507-ff5afc07dd28" (UID: "203f17dc-dd8b-42dc-9507-ff5afc07dd28"). InnerVolumeSpecName "kube-api-access-vzftx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:50.098965 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.098938    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:50.098965 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.098964    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzftx\" (UniqueName: \"kubernetes.io/projected/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kube-api-access-vzftx\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:50.099161 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.098978    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/203f17dc-dd8b-42dc-9507-ff5afc07dd28-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:50.099161 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.098991    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/203f17dc-dd8b-42dc-9507-ff5afc07dd28-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:50.099161 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.099004    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/203f17dc-dd8b-42dc-9507-ff5afc07dd28-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:50.742783 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.742749    4511 generic.go:356] "Generic (PLEG): container finished" podID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerID="3bfd54a5a0d2ea20ed91ef6b493556fc655ef8b04714ce845ea8c8dda26479e7" exitCode=0
Oct 30 19:22:50.743419 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.742827    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"
Oct 30 19:22:50.743419 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.742837    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" event={"ID":"203f17dc-dd8b-42dc-9507-ff5afc07dd28","Type":"ContainerDied","Data":"3bfd54a5a0d2ea20ed91ef6b493556fc655ef8b04714ce845ea8c8dda26479e7"}
Oct 30 19:22:50.743419 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.742894    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4" event={"ID":"203f17dc-dd8b-42dc-9507-ff5afc07dd28","Type":"ContainerDied","Data":"5b083036d77c36751f50dc8416156a56d80f852b6da9d836de7299439c484bfb"}
Oct 30 19:22:50.743419 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.742921    4511 scope.go:122] "RemoveContainer" containerID="302cd993ef43127216d28eab9ea7acc368f3aa68379d2031a7877328dec34385"
Oct 30 19:22:50.751837 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.751817    4511 scope.go:122] "RemoveContainer" containerID="3bfd54a5a0d2ea20ed91ef6b493556fc655ef8b04714ce845ea8c8dda26479e7"
Oct 30 19:22:50.759062 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.759044    4511 scope.go:122] "RemoveContainer" containerID="c4d63583697f39602ad859747178f3766e712e412e0f1bc6c4d26de5243d4365"
Oct 30 19:22:50.765957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.765941    4511 scope.go:122] "RemoveContainer" containerID="302cd993ef43127216d28eab9ea7acc368f3aa68379d2031a7877328dec34385"
Oct 30 19:22:50.766209 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:22:50.766185    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"302cd993ef43127216d28eab9ea7acc368f3aa68379d2031a7877328dec34385\": container with ID starting with 302cd993ef43127216d28eab9ea7acc368f3aa68379d2031a7877328dec34385 not found: ID does not exist" containerID="302cd993ef43127216d28eab9ea7acc368f3aa68379d2031a7877328dec34385"
Oct 30 19:22:50.766283 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.766222    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"302cd993ef43127216d28eab9ea7acc368f3aa68379d2031a7877328dec34385"} err="failed to get container status \"302cd993ef43127216d28eab9ea7acc368f3aa68379d2031a7877328dec34385\": rpc error: code = NotFound desc = could not find container \"302cd993ef43127216d28eab9ea7acc368f3aa68379d2031a7877328dec34385\": container with ID starting with 302cd993ef43127216d28eab9ea7acc368f3aa68379d2031a7877328dec34385 not found: ID does not exist"
Oct 30 19:22:50.766283 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.766246    4511 scope.go:122] "RemoveContainer" containerID="3bfd54a5a0d2ea20ed91ef6b493556fc655ef8b04714ce845ea8c8dda26479e7"
Oct 30 19:22:50.766489 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:22:50.766466    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bfd54a5a0d2ea20ed91ef6b493556fc655ef8b04714ce845ea8c8dda26479e7\": container with ID starting with 3bfd54a5a0d2ea20ed91ef6b493556fc655ef8b04714ce845ea8c8dda26479e7 not found: ID does not exist" containerID="3bfd54a5a0d2ea20ed91ef6b493556fc655ef8b04714ce845ea8c8dda26479e7"
Oct 30 19:22:50.766532 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.766490    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bfd54a5a0d2ea20ed91ef6b493556fc655ef8b04714ce845ea8c8dda26479e7"} err="failed to get container status \"3bfd54a5a0d2ea20ed91ef6b493556fc655ef8b04714ce845ea8c8dda26479e7\": rpc error: code = NotFound desc = could not find container \"3bfd54a5a0d2ea20ed91ef6b493556fc655ef8b04714ce845ea8c8dda26479e7\": container with ID starting with 3bfd54a5a0d2ea20ed91ef6b493556fc655ef8b04714ce845ea8c8dda26479e7 not found: ID does not exist"
Oct 30 19:22:50.766532 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.766503    4511 scope.go:122] "RemoveContainer" containerID="c4d63583697f39602ad859747178f3766e712e412e0f1bc6c4d26de5243d4365"
Oct 30 19:22:50.766715 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:22:50.766699    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d63583697f39602ad859747178f3766e712e412e0f1bc6c4d26de5243d4365\": container with ID starting with c4d63583697f39602ad859747178f3766e712e412e0f1bc6c4d26de5243d4365 not found: ID does not exist" containerID="c4d63583697f39602ad859747178f3766e712e412e0f1bc6c4d26de5243d4365"
Oct 30 19:22:50.766781 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.766719    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d63583697f39602ad859747178f3766e712e412e0f1bc6c4d26de5243d4365"} err="failed to get container status \"c4d63583697f39602ad859747178f3766e712e412e0f1bc6c4d26de5243d4365\": rpc error: code = NotFound desc = could not find container \"c4d63583697f39602ad859747178f3766e712e412e0f1bc6c4d26de5243d4365\": container with ID starting with c4d63583697f39602ad859747178f3766e712e412e0f1bc6c4d26de5243d4365 not found: ID does not exist"
Oct 30 19:22:50.949093 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.949068    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"]
Oct 30 19:22:50.953464 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:50.953447    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-predictor-58b554b449-nnlv4"]
Oct 30 19:22:51.656375 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:22:51.656342    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" path="/var/lib/kubelet/pods/203f17dc-dd8b-42dc-9507-ff5afc07dd28/volumes"
Oct 30 19:23:10.156199 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:23:10.156162    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:23:25.837519 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:25.837415    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 19:23:25.846211 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:25.846192    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 19:23:27.661484 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.661454    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"]
Oct 30 19:23:27.661863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.661734    4511 cpu_manager.go:414] "containerMap: removing container" podUID="7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" containerName="kserve-container"
Oct 30 19:23:27.661863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.661745    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" containerName="kserve-container"
Oct 30 19:23:27.661863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.661753    4511 cpu_manager.go:414] "containerMap: removing container" podUID="7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" containerName="kube-rbac-proxy"
Oct 30 19:23:27.661863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.661758    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" containerName="kube-rbac-proxy"
Oct 30 19:23:27.661863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.661766    4511 cpu_manager.go:414] "containerMap: removing container" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kserve-container"
Oct 30 19:23:27.661863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.661771    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kserve-container"
Oct 30 19:23:27.661863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.661778    4511 cpu_manager.go:414] "containerMap: removing container" podUID="7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" containerName="storage-initializer"
Oct 30 19:23:27.661863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.661782    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" containerName="storage-initializer"
Oct 30 19:23:27.661863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.661795    4511 cpu_manager.go:414] "containerMap: removing container" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="storage-initializer"
Oct 30 19:23:27.661863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.661800    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="storage-initializer"
Oct 30 19:23:27.661863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.661813    4511 cpu_manager.go:414] "containerMap: removing container" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kube-rbac-proxy"
Oct 30 19:23:27.661863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.661817    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kube-rbac-proxy"
Oct 30 19:23:27.661863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.661861    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kube-rbac-proxy"
Oct 30 19:23:27.661863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.661870    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" containerName="kserve-container"
Oct 30 19:23:27.662301 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.661876    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bcc90a9-ec21-482e-8a7c-db42c3e03e0c" containerName="kube-rbac-proxy"
Oct 30 19:23:27.662301 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.661882    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="203f17dc-dd8b-42dc-9507-ff5afc07dd28" containerName="kserve-container"
Oct 30 19:23:27.666163 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.666119    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:27.719877 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.719849    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:23:27.720012 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.719898    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:23:27.720012 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.719952    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:23:27.720012 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.719976    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:23:27.767278 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.767258    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Oct 30 19:23:27.767462 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.767325    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:23:27.767462 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.767338    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:23:27.768837 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.768820    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"]
Oct 30 19:23:27.782000 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.781969    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:27.782118 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.782009    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:27.782229 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.782204    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:27.782290 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.782268    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:27.782369 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.782351    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khhh7\" (UniqueName: \"kubernetes.io/projected/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kube-api-access-khhh7\") pod \"isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:27.882675 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.882648    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:27.882789 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.882695    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:27.882789 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.882762    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khhh7\" (UniqueName: \"kubernetes.io/projected/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kube-api-access-khhh7\") pod \"isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:27.882907 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.882885    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:27.882952 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.882932    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:27.883418 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.883387    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:27.883546 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.883483    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:27.883546 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.883493    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:27.885236 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.885212    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:27.943263 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.943206    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khhh7\" (UniqueName: \"kubernetes.io/projected/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kube-api-access-khhh7\") pod \"isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:27.985121 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:27.985097    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:28.406653 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:28.406621    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"]
Oct 30 19:23:28.412392 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:23:28.412362    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7056ca_dfa2_4c22_9389_2c41ac5761f7.slice/crio-d3e7e868fdd9791098845249e36207bfc40b16fa1408c669937c53bdcf2d8ed8 WatchSource:0}: Error finding container d3e7e868fdd9791098845249e36207bfc40b16fa1408c669937c53bdcf2d8ed8: Status 404 returned error can't find the container with id d3e7e868fdd9791098845249e36207bfc40b16fa1408c669937c53bdcf2d8ed8
Oct 30 19:23:28.890451 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:28.890414    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg" event={"ID":"2c7056ca-dfa2-4c22-9389-2c41ac5761f7","Type":"ContainerStarted","Data":"3ad5f3d667a5bf430c2753fbb29074ce267c83380900406639a8b6d79a6adc0c"}
Oct 30 19:23:28.890887 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:28.890483    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg" event={"ID":"2c7056ca-dfa2-4c22-9389-2c41ac5761f7","Type":"ContainerStarted","Data":"d3e7e868fdd9791098845249e36207bfc40b16fa1408c669937c53bdcf2d8ed8"}
Oct 30 19:23:32.907570 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:32.907543    4511 generic.go:356] "Generic (PLEG): container finished" podID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" containerID="3ad5f3d667a5bf430c2753fbb29074ce267c83380900406639a8b6d79a6adc0c" exitCode=0
Oct 30 19:23:32.907957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:32.907605    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg" event={"ID":"2c7056ca-dfa2-4c22-9389-2c41ac5761f7","Type":"ContainerDied","Data":"3ad5f3d667a5bf430c2753fbb29074ce267c83380900406639a8b6d79a6adc0c"}
Oct 30 19:23:33.914578 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:33.914532    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg" event={"ID":"2c7056ca-dfa2-4c22-9389-2c41ac5761f7","Type":"ContainerStarted","Data":"fc928c375310b1098bf7895f96be2f425fc3a887c48e29af459f39b82d13322e"}
Oct 30 19:23:33.914963 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:33.914602    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg" event={"ID":"2c7056ca-dfa2-4c22-9389-2c41ac5761f7","Type":"ContainerStarted","Data":"ffbe756012f8a454ce76331dfc81d59febf3023c6200ff3d7ac790e1b80afb85"}
Oct 30 19:23:33.914963 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:33.914891    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:34.062085 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:34.062028    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg" podStartSLOduration=3.089202346 podStartE2EDuration="7.06201686s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.972814514s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:34.030374253 +0000 UTC m=+2708.548059153" watchObservedRunningTime="2026-10-30 19:23:34.06201686 +0000 UTC m=+2708.579701759"
Oct 30 19:23:34.918900 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:34.918860    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:23:40.156564 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:23:40.156532    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:23:40.928481 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:23:40.928450    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:24:10.156482 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:24:10.156447    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:24:11.016776 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:11.016732    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg" podUID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 30 19:24:15.444009 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:15.443973    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"]
Oct 30 19:24:15.444544 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:15.444293    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kserve-container" containerID="cri-o://933ab855192afac1b36a2eb9f799c1e19f5f81c2041a83b1fa10f62e70ec904e" gracePeriod=30
Oct 30 19:24:15.444544 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:15.444338    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kube-rbac-proxy" containerID="cri-o://4aad789501aa370132e51c3191a7b3cee579d1b06f0bfe08947a6d425955c441" gracePeriod=30
Oct 30 19:24:16.079960 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:16.079929    4511 generic.go:356] "Generic (PLEG): container finished" podID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerID="4aad789501aa370132e51c3191a7b3cee579d1b06f0bfe08947a6d425955c441" exitCode=0
Oct 30 19:24:16.080155 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:16.080012    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" event={"ID":"77f553bc-36b5-47bf-a08d-8c7e8aacd02e","Type":"ContainerDied","Data":"4aad789501aa370132e51c3191a7b3cee579d1b06f0bfe08947a6d425955c441"}
Oct 30 19:24:17.380894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:17.380853    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Oct 30 19:24:18.652013 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:18.651967    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Oct 30 19:24:19.983930 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:19.983900    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:24:20.078656 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.078593    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kube-api-access-6n6w5\" (UniqueName: \"kubernetes.io/projected/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kube-api-access-6n6w5\") pod \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") "
Oct 30 19:24:20.078656 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.078637    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-cabundle-cert\") pod \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") "
Oct 30 19:24:20.078656 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.078658    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kserve-provision-location\") pod \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") "
Oct 30 19:24:20.078887 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.078709    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-proxy-tls\" (UniqueName: \"kubernetes.io/secret/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-proxy-tls\") pod \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") "
Oct 30 19:24:20.078887 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.078732    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kube-rbac-proxy-sar-config\") pod \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\" (UID: \"77f553bc-36b5-47bf-a08d-8c7e8aacd02e\") "
Oct 30 19:24:20.079031 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.079005    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-cabundle-cert" pod "77f553bc-36b5-47bf-a08d-8c7e8aacd02e" (UID: "77f553bc-36b5-47bf-a08d-8c7e8aacd02e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:20.079081 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.079041    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kserve-provision-location" pod "77f553bc-36b5-47bf-a08d-8c7e8aacd02e" (UID: "77f553bc-36b5-47bf-a08d-8c7e8aacd02e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:20.079144 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.079112    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kube-rbac-proxy-sar-config" pod "77f553bc-36b5-47bf-a08d-8c7e8aacd02e" (UID: "77f553bc-36b5-47bf-a08d-8c7e8aacd02e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:20.080770 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.080744    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kube-api-access-6n6w5" pod "77f553bc-36b5-47bf-a08d-8c7e8aacd02e" (UID: "77f553bc-36b5-47bf-a08d-8c7e8aacd02e"). InnerVolumeSpecName "kube-api-access-6n6w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:20.080852 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.080826    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-proxy-tls" pod "77f553bc-36b5-47bf-a08d-8c7e8aacd02e" (UID: "77f553bc-36b5-47bf-a08d-8c7e8aacd02e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:20.097817 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.097788    4511 generic.go:356] "Generic (PLEG): container finished" podID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerID="933ab855192afac1b36a2eb9f799c1e19f5f81c2041a83b1fa10f62e70ec904e" exitCode=0
Oct 30 19:24:20.097897 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.097845    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"
Oct 30 19:24:20.097897 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.097865    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" event={"ID":"77f553bc-36b5-47bf-a08d-8c7e8aacd02e","Type":"ContainerDied","Data":"933ab855192afac1b36a2eb9f799c1e19f5f81c2041a83b1fa10f62e70ec904e"}
Oct 30 19:24:20.097971 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.097924    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw" event={"ID":"77f553bc-36b5-47bf-a08d-8c7e8aacd02e","Type":"ContainerDied","Data":"3fa48241695b7bde4f61d244c1f354205bf1c5667c04407565ce042f7332e51a"}
Oct 30 19:24:20.097971 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.097952    4511 scope.go:122] "RemoveContainer" containerID="4aad789501aa370132e51c3191a7b3cee579d1b06f0bfe08947a6d425955c441"
Oct 30 19:24:20.106894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.106872    4511 scope.go:122] "RemoveContainer" containerID="933ab855192afac1b36a2eb9f799c1e19f5f81c2041a83b1fa10f62e70ec904e"
Oct 30 19:24:20.113862 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.113848    4511 scope.go:122] "RemoveContainer" containerID="2096e70790b2b9f77ac72bb6c1bc5cdf196732caccdd28d48159589903b09420"
Oct 30 19:24:20.122157 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.122122    4511 scope.go:122] "RemoveContainer" containerID="4aad789501aa370132e51c3191a7b3cee579d1b06f0bfe08947a6d425955c441"
Oct 30 19:24:20.122598 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:24:20.122576    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aad789501aa370132e51c3191a7b3cee579d1b06f0bfe08947a6d425955c441\": container with ID starting with 4aad789501aa370132e51c3191a7b3cee579d1b06f0bfe08947a6d425955c441 not found: ID does not exist" containerID="4aad789501aa370132e51c3191a7b3cee579d1b06f0bfe08947a6d425955c441"
Oct 30 19:24:20.122703 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.122617    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aad789501aa370132e51c3191a7b3cee579d1b06f0bfe08947a6d425955c441"} err="failed to get container status \"4aad789501aa370132e51c3191a7b3cee579d1b06f0bfe08947a6d425955c441\": rpc error: code = NotFound desc = could not find container \"4aad789501aa370132e51c3191a7b3cee579d1b06f0bfe08947a6d425955c441\": container with ID starting with 4aad789501aa370132e51c3191a7b3cee579d1b06f0bfe08947a6d425955c441 not found: ID does not exist"
Oct 30 19:24:20.122703 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.122642    4511 scope.go:122] "RemoveContainer" containerID="933ab855192afac1b36a2eb9f799c1e19f5f81c2041a83b1fa10f62e70ec904e"
Oct 30 19:24:20.122956 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:24:20.122923    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933ab855192afac1b36a2eb9f799c1e19f5f81c2041a83b1fa10f62e70ec904e\": container with ID starting with 933ab855192afac1b36a2eb9f799c1e19f5f81c2041a83b1fa10f62e70ec904e not found: ID does not exist" containerID="933ab855192afac1b36a2eb9f799c1e19f5f81c2041a83b1fa10f62e70ec904e"
Oct 30 19:24:20.123044 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.122962    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933ab855192afac1b36a2eb9f799c1e19f5f81c2041a83b1fa10f62e70ec904e"} err="failed to get container status \"933ab855192afac1b36a2eb9f799c1e19f5f81c2041a83b1fa10f62e70ec904e\": rpc error: code = NotFound desc = could not find container \"933ab855192afac1b36a2eb9f799c1e19f5f81c2041a83b1fa10f62e70ec904e\": container with ID starting with 933ab855192afac1b36a2eb9f799c1e19f5f81c2041a83b1fa10f62e70ec904e not found: ID does not exist"
Oct 30 19:24:20.123044 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.122983    4511 scope.go:122] "RemoveContainer" containerID="2096e70790b2b9f77ac72bb6c1bc5cdf196732caccdd28d48159589903b09420"
Oct 30 19:24:20.123389 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:24:20.123300    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2096e70790b2b9f77ac72bb6c1bc5cdf196732caccdd28d48159589903b09420\": container with ID starting with 2096e70790b2b9f77ac72bb6c1bc5cdf196732caccdd28d48159589903b09420 not found: ID does not exist" containerID="2096e70790b2b9f77ac72bb6c1bc5cdf196732caccdd28d48159589903b09420"
Oct 30 19:24:20.123389 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.123327    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2096e70790b2b9f77ac72bb6c1bc5cdf196732caccdd28d48159589903b09420"} err="failed to get container status \"2096e70790b2b9f77ac72bb6c1bc5cdf196732caccdd28d48159589903b09420\": rpc error: code = NotFound desc = could not find container \"2096e70790b2b9f77ac72bb6c1bc5cdf196732caccdd28d48159589903b09420\": container with ID starting with 2096e70790b2b9f77ac72bb6c1bc5cdf196732caccdd28d48159589903b09420 not found: ID does not exist"
Oct 30 19:24:20.179985 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.179961    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:20.179985 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.179983    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:20.180146 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.179995    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n6w5\" (UniqueName: \"kubernetes.io/projected/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kube-api-access-6n6w5\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:20.180146 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.180003    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:20.180146 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.180012    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77f553bc-36b5-47bf-a08d-8c7e8aacd02e-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:20.254743 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.254717    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"]
Oct 30 19:24:20.258923 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.258904    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-promote-v2-predictor-76656f7564-kwkjw"]
Oct 30 19:24:20.931663 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:20.931631    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:24:21.657688 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:21.657646    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" path="/var/lib/kubelet/pods/77f553bc-36b5-47bf-a08d-8c7e8aacd02e/volumes"
Oct 30 19:24:24.550119 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.550089    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"]
Oct 30 19:24:24.550501 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.550402    4511 cpu_manager.go:414] "containerMap: removing container" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kserve-container"
Oct 30 19:24:24.550501 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.550413    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kserve-container"
Oct 30 19:24:24.550501 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.550430    4511 cpu_manager.go:414] "containerMap: removing container" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="storage-initializer"
Oct 30 19:24:24.550501 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.550436    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="storage-initializer"
Oct 30 19:24:24.550501 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.550444    4511 cpu_manager.go:414] "containerMap: removing container" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kube-rbac-proxy"
Oct 30 19:24:24.550501 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.550450    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kube-rbac-proxy"
Oct 30 19:24:24.550501 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.550497    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kserve-container"
Oct 30 19:24:24.550501 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.550506    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f553bc-36b5-47bf-a08d-8c7e8aacd02e" containerName="kube-rbac-proxy"
Oct 30 19:24:24.553660 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.553643    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:24.604349 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.604312    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:24:24.604349 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.604326    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:24:24.604349 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.604315    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:24:24.604349 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.604324    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Oct 30 19:24:24.612646 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.612624    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb826157-08bd-4622-a2c6-b6ba1240601a-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:24.612724 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.612661    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb826157-08bd-4622-a2c6-b6ba1240601a-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:24.612782 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.612741    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb826157-08bd-4622-a2c6-b6ba1240601a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:24.612838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.612787    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb826157-08bd-4622-a2c6-b6ba1240601a-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:24.612838 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.612820    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwlrk\" (UniqueName: \"kubernetes.io/projected/eb826157-08bd-4622-a2c6-b6ba1240601a-kube-api-access-vwlrk\") pod \"isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:24.651607 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.651563    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:24:24.651607 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.651574    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:24:24.651607 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.651577    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Oct 30 19:24:24.653356 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.653337    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"]
Oct 30 19:24:24.713534 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.713492    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb826157-08bd-4622-a2c6-b6ba1240601a-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:24.713676 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.713544    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb826157-08bd-4622-a2c6-b6ba1240601a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:24.713757 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.713666    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb826157-08bd-4622-a2c6-b6ba1240601a-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:24.713757 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.713715    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwlrk\" (UniqueName: \"kubernetes.io/projected/eb826157-08bd-4622-a2c6-b6ba1240601a-kube-api-access-vwlrk\") pod \"isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:24.713860 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.713790    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb826157-08bd-4622-a2c6-b6ba1240601a-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:24.713922 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.713873    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb826157-08bd-4622-a2c6-b6ba1240601a-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:24.714185 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.714165    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb826157-08bd-4622-a2c6-b6ba1240601a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:24.714392 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.714372    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb826157-08bd-4622-a2c6-b6ba1240601a-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:24.716181 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.716165    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb826157-08bd-4622-a2c6-b6ba1240601a-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:24.772307 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.772288    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwlrk\" (UniqueName: \"kubernetes.io/projected/eb826157-08bd-4622-a2c6-b6ba1240601a-kube-api-access-vwlrk\") pod \"isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:24.871311 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:24.871256    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:25.293035 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.292960    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"]
Oct 30 19:24:25.299526 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:24:25.299499    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb826157_08bd_4622_a2c6_b6ba1240601a.slice/crio-04a1193a0706f0b99f47be241ff284891051da47bbde872d91ee06e210391a49 WatchSource:0}: Error finding container 04a1193a0706f0b99f47be241ff284891051da47bbde872d91ee06e210391a49: Status 404 returned error can't find the container with id 04a1193a0706f0b99f47be241ff284891051da47bbde872d91ee06e210391a49
Oct 30 19:24:25.513568 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.513527    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"]
Oct 30 19:24:25.517788 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.517766    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:25.568322 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.568288    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-rollback-predictor-serving-cert\""
Oct 30 19:24:25.576617 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.576589    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"]
Oct 30 19:24:25.620194 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.620165    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-46lcv\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:25.620323 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.620205    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-46lcv\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:25.620323 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.620302    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77df7a1b-02b2-48df-809f-e3ca34d79ef1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-46lcv\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:25.620458 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.620438    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77df7a1b-02b2-48df-809f-e3ca34d79ef1-proxy-tls\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-46lcv\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:25.620502 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.620477    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkp8k\" (UniqueName: \"kubernetes.io/projected/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kube-api-access-tkp8k\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-46lcv\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:25.720920 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.720886    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77df7a1b-02b2-48df-809f-e3ca34d79ef1-proxy-tls\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-46lcv\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:25.721087 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.720948    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkp8k\" (UniqueName: \"kubernetes.io/projected/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kube-api-access-tkp8k\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-46lcv\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:25.721087 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.720988    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-46lcv\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:25.721087 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.721027    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-46lcv\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:25.721313 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.721112    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77df7a1b-02b2-48df-809f-e3ca34d79ef1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-46lcv\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:25.721515 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.721490    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-46lcv\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:25.721721 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.721702    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-46lcv\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:25.721889 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.721869    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77df7a1b-02b2-48df-809f-e3ca34d79ef1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-46lcv\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:25.770301 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.770278    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-rollback-predictor-serving-cert\""
Oct 30 19:24:25.773500 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.773465    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77df7a1b-02b2-48df-809f-e3ca34d79ef1-proxy-tls\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-46lcv\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:25.780559 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.780538    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkp8k\" (UniqueName: \"kubernetes.io/projected/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kube-api-access-tkp8k\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-46lcv\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:25.837723 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:25.837666    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:26.125015 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:26.124948    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" event={"ID":"eb826157-08bd-4622-a2c6-b6ba1240601a","Type":"ContainerStarted","Data":"8043aecd1582d59595beb0cec97e56e2c3ac61329e92e769c1f9fd02e3e42efd"}
Oct 30 19:24:26.125015 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:26.125009    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" event={"ID":"eb826157-08bd-4622-a2c6-b6ba1240601a","Type":"ContainerStarted","Data":"04a1193a0706f0b99f47be241ff284891051da47bbde872d91ee06e210391a49"}
Oct 30 19:24:26.273389 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:24:26.273362    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77df7a1b_02b2_48df_809f_e3ca34d79ef1.slice/crio-c8248cb74cbde688daf15100a927c46d99595f8083f61b592df770ff88437f42 WatchSource:0}: Error finding container c8248cb74cbde688daf15100a927c46d99595f8083f61b592df770ff88437f42: Status 404 returned error can't find the container with id c8248cb74cbde688daf15100a927c46d99595f8083f61b592df770ff88437f42
Oct 30 19:24:26.289180 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:26.289160    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"]
Oct 30 19:24:27.130380 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:27.130341    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" event={"ID":"77df7a1b-02b2-48df-809f-e3ca34d79ef1","Type":"ContainerStarted","Data":"b9b339d2102ae7eee0b3f5711db6741da3a70f4b8e91d0047b0b0fce34a13eec"}
Oct 30 19:24:27.130799 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:27.130401    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" event={"ID":"77df7a1b-02b2-48df-809f-e3ca34d79ef1","Type":"ContainerStarted","Data":"c8248cb74cbde688daf15100a927c46d99595f8083f61b592df770ff88437f42"}
Oct 30 19:24:29.064626 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:29.064594    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"]
Oct 30 19:24:29.065020 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:29.064806    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg" podUID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" containerName="kserve-container" containerID="cri-o://ffbe756012f8a454ce76331dfc81d59febf3023c6200ff3d7ac790e1b80afb85" gracePeriod=30
Oct 30 19:24:29.065020 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:29.064844    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg" podUID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" containerName="kube-rbac-proxy" containerID="cri-o://fc928c375310b1098bf7895f96be2f425fc3a887c48e29af459f39b82d13322e" gracePeriod=30
Oct 30 19:24:29.145402 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:29.145367    4511 generic.go:356] "Generic (PLEG): container finished" podID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerID="8043aecd1582d59595beb0cec97e56e2c3ac61329e92e769c1f9fd02e3e42efd" exitCode=0
Oct 30 19:24:29.145538 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:29.145426    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" event={"ID":"eb826157-08bd-4622-a2c6-b6ba1240601a","Type":"ContainerDied","Data":"8043aecd1582d59595beb0cec97e56e2c3ac61329e92e769c1f9fd02e3e42efd"}
Oct 30 19:24:30.151664 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:30.151637    4511 generic.go:356] "Generic (PLEG): container finished" podID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" containerID="fc928c375310b1098bf7895f96be2f425fc3a887c48e29af459f39b82d13322e" exitCode=0
Oct 30 19:24:30.152057 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:30.151711    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg" event={"ID":"2c7056ca-dfa2-4c22-9389-2c41ac5761f7","Type":"ContainerDied","Data":"fc928c375310b1098bf7895f96be2f425fc3a887c48e29af459f39b82d13322e"}
Oct 30 19:24:30.154018 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:30.153994    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" event={"ID":"eb826157-08bd-4622-a2c6-b6ba1240601a","Type":"ContainerStarted","Data":"c4aaafe9298b8ea906250b654e461bf6b4489de98a62997e482e5c153e5ccb8a"}
Oct 30 19:24:30.154136 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:30.154050    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" event={"ID":"eb826157-08bd-4622-a2c6-b6ba1240601a","Type":"ContainerStarted","Data":"c0430983628d03d9649debde7378da0343634d7f5fb29defd9c752363f4e118a"}
Oct 30 19:24:30.154388 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:30.154364    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:30.154493 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:30.154417    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:30.156012 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:30.155992    4511 generic.go:356] "Generic (PLEG): container finished" podID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerID="b9b339d2102ae7eee0b3f5711db6741da3a70f4b8e91d0047b0b0fce34a13eec" exitCode=0
Oct 30 19:24:30.156108 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:30.156008    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 30 19:24:30.156108 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:30.156032    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" event={"ID":"77df7a1b-02b2-48df-809f-e3ca34d79ef1","Type":"ContainerDied","Data":"b9b339d2102ae7eee0b3f5711db6741da3a70f4b8e91d0047b0b0fce34a13eec"}
Oct 30 19:24:30.270662 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:30.270616    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" podStartSLOduration=2.693165855 podStartE2EDuration="6.270602695s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.57743684s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:30.269577701 +0000 UTC m=+2764.787262617" watchObservedRunningTime="2026-10-30 19:24:30.270602695 +0000 UTC m=+2764.788287595"
Oct 30 19:24:30.923070 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:30.923021    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg" podUID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Oct 30 19:24:31.161980 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:31.161943    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" event={"ID":"77df7a1b-02b2-48df-809f-e3ca34d79ef1","Type":"ContainerStarted","Data":"68563aec61a0038b0e5e495c5a430b0276f4129ea9d64e45f101834d548a0190"}
Oct 30 19:24:31.162497 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:31.162000    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" event={"ID":"77df7a1b-02b2-48df-809f-e3ca34d79ef1","Type":"ContainerStarted","Data":"fd84b53f1547e1b0c4e69e0b10a099d853247454716e406899dbcc938ab3c008"}
Oct 30 19:24:31.162497 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:31.162202    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 30 19:24:31.162627 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:31.162594    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:31.162705 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:31.162689    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:31.163933 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:31.163912    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 30 19:24:31.277250 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:31.277166    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" podStartSLOduration=2.514684951 podStartE2EDuration="6.277153704s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.762468753s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:31.276479811 +0000 UTC m=+2765.794164711" watchObservedRunningTime="2026-10-30 19:24:31.277153704 +0000 UTC m=+2765.794838598"
Oct 30 19:24:31.970301 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:31.970259    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg" podUID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.49:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:32.166522 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:32.166470    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 30 19:24:35.923996 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:35.923896    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg" podUID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Oct 30 19:24:36.166748 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.166720    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:24:36.167187 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.167162    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 30 19:24:36.184265 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.184240    4511 generic.go:356] "Generic (PLEG): container finished" podID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" containerID="ffbe756012f8a454ce76331dfc81d59febf3023c6200ff3d7ac790e1b80afb85" exitCode=0
Oct 30 19:24:36.184384 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.184294    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg" event={"ID":"2c7056ca-dfa2-4c22-9389-2c41ac5761f7","Type":"ContainerDied","Data":"ffbe756012f8a454ce76331dfc81d59febf3023c6200ff3d7ac790e1b80afb85"}
Oct 30 19:24:36.303603 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.303583    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:24:36.406668 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.406644    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kube-api-access-khhh7\" (UniqueName: \"kubernetes.io/projected/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kube-api-access-khhh7\") pod \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") "
Oct 30 19:24:36.406824 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.406719    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-proxy-tls\") pod \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") "
Oct 30 19:24:36.406824 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.406746    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kube-rbac-proxy-sar-config\") pod \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") "
Oct 30 19:24:36.406824 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.406786    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kserve-provision-location\") pod \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") "
Oct 30 19:24:36.406824 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.406814    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-cabundle-cert\") pod \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\" (UID: \"2c7056ca-dfa2-4c22-9389-2c41ac5761f7\") "
Oct 30 19:24:36.407110 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.407083    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kserve-provision-location" pod "2c7056ca-dfa2-4c22-9389-2c41ac5761f7" (UID: "2c7056ca-dfa2-4c22-9389-2c41ac5761f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:36.407196 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.407150    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kube-rbac-proxy-sar-config" pod "2c7056ca-dfa2-4c22-9389-2c41ac5761f7" (UID: "2c7056ca-dfa2-4c22-9389-2c41ac5761f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:36.407233 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.407198    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-cabundle-cert" pod "2c7056ca-dfa2-4c22-9389-2c41ac5761f7" (UID: "2c7056ca-dfa2-4c22-9389-2c41ac5761f7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:36.408874 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.408854    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-proxy-tls" pod "2c7056ca-dfa2-4c22-9389-2c41ac5761f7" (UID: "2c7056ca-dfa2-4c22-9389-2c41ac5761f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:36.408982 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.408965    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kube-api-access-khhh7" pod "2c7056ca-dfa2-4c22-9389-2c41ac5761f7" (UID: "2c7056ca-dfa2-4c22-9389-2c41ac5761f7"). InnerVolumeSpecName "kube-api-access-khhh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:36.508284 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.508214    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khhh7\" (UniqueName: \"kubernetes.io/projected/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kube-api-access-khhh7\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:36.508284 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.508240    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:36.508284 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.508253    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:36.508284 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.508266    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:36.508284 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:36.508279    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c7056ca-dfa2-4c22-9389-2c41ac5761f7-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:37.171590 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:37.171560    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:24:37.171988 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:37.171966    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 30 19:24:37.190320 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:37.190290    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg" event={"ID":"2c7056ca-dfa2-4c22-9389-2c41ac5761f7","Type":"ContainerDied","Data":"d3e7e868fdd9791098845249e36207bfc40b16fa1408c669937c53bdcf2d8ed8"}
Oct 30 19:24:37.190482 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:37.190338    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"
Oct 30 19:24:37.190482 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:37.190358    4511 scope.go:122] "RemoveContainer" containerID="fc928c375310b1098bf7895f96be2f425fc3a887c48e29af459f39b82d13322e"
Oct 30 19:24:37.198955 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:37.198936    4511 scope.go:122] "RemoveContainer" containerID="ffbe756012f8a454ce76331dfc81d59febf3023c6200ff3d7ac790e1b80afb85"
Oct 30 19:24:37.206063 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:37.206046    4511 scope.go:122] "RemoveContainer" containerID="3ad5f3d667a5bf430c2753fbb29074ce267c83380900406639a8b6d79a6adc0c"
Oct 30 19:24:37.394178 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:37.394144    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"]
Oct 30 19:24:37.401408 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:37.401386    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-runtime-predictor-6949b58565-9z7lg"]
Oct 30 19:24:37.657548 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:37.657513    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" path="/var/lib/kubelet/pods/2c7056ca-dfa2-4c22-9389-2c41ac5761f7/volumes"
Oct 30 19:24:39.925962 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:39.925932    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"]
Oct 30 19:24:39.926351 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:39.926239    4511 cpu_manager.go:414] "containerMap: removing container" podUID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" containerName="kserve-container"
Oct 30 19:24:39.926351 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:39.926250    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" containerName="kserve-container"
Oct 30 19:24:39.926351 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:39.926275    4511 cpu_manager.go:414] "containerMap: removing container" podUID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" containerName="storage-initializer"
Oct 30 19:24:39.926351 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:39.926281    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" containerName="storage-initializer"
Oct 30 19:24:39.926351 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:39.926291    4511 cpu_manager.go:414] "containerMap: removing container" podUID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" containerName="kube-rbac-proxy"
Oct 30 19:24:39.926351 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:39.926296    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" containerName="kube-rbac-proxy"
Oct 30 19:24:39.926351 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:39.926342    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" containerName="kube-rbac-proxy"
Oct 30 19:24:39.926351 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:39.926350    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c7056ca-dfa2-4c22-9389-2c41ac5761f7" containerName="kserve-container"
Oct 30 19:24:39.929571 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:39.929551    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:24:40.027879 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.027855    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:24:40.028013 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.027919    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:24:40.028090 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.028073    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Oct 30 19:24:40.028173 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.028075    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:24:40.028173 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.028082    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:24:40.028173 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.028119    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:24:40.028173 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.028139    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Oct 30 19:24:40.031600 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.031579    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"]
Oct 30 19:24:40.035202 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.035180    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80c65b8e-8533-469e-bb66-50f25eaef7e9-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-7d5fcf856-nj9pj\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:24:40.035327 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.035215    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80c65b8e-8533-469e-bb66-50f25eaef7e9-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-7d5fcf856-nj9pj\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:24:40.035405 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.035382    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80c65b8e-8533-469e-bb66-50f25eaef7e9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-7d5fcf856-nj9pj\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:24:40.035458 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.035434    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5sp9\" (UniqueName: \"kubernetes.io/projected/80c65b8e-8533-469e-bb66-50f25eaef7e9-kube-api-access-h5sp9\") pod \"isvc-xgboost-v2-predictor-7d5fcf856-nj9pj\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:24:40.035498 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.035483    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80c65b8e-8533-469e-bb66-50f25eaef7e9-proxy-tls\") pod \"isvc-xgboost-v2-predictor-7d5fcf856-nj9pj\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:24:40.136197 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.136173    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80c65b8e-8533-469e-bb66-50f25eaef7e9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-7d5fcf856-nj9pj\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:24:40.136345 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.136209    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5sp9\" (UniqueName: \"kubernetes.io/projected/80c65b8e-8533-469e-bb66-50f25eaef7e9-kube-api-access-h5sp9\") pod \"isvc-xgboost-v2-predictor-7d5fcf856-nj9pj\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:24:40.136345 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.136228    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80c65b8e-8533-469e-bb66-50f25eaef7e9-proxy-tls\") pod \"isvc-xgboost-v2-predictor-7d5fcf856-nj9pj\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:24:40.136345 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.136280    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80c65b8e-8533-469e-bb66-50f25eaef7e9-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-7d5fcf856-nj9pj\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:24:40.136345 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.136302    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80c65b8e-8533-469e-bb66-50f25eaef7e9-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-7d5fcf856-nj9pj\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:24:40.136706 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.136683    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80c65b8e-8533-469e-bb66-50f25eaef7e9-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-7d5fcf856-nj9pj\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:24:40.136943 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.136926    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80c65b8e-8533-469e-bb66-50f25eaef7e9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-7d5fcf856-nj9pj\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:24:40.137043 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.137023    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80c65b8e-8533-469e-bb66-50f25eaef7e9-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-7d5fcf856-nj9pj\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:24:40.138788 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.138764    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80c65b8e-8533-469e-bb66-50f25eaef7e9-proxy-tls\") pod \"isvc-xgboost-v2-predictor-7d5fcf856-nj9pj\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:24:40.156978 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:24:40.156951    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:24:40.195461 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.195398    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5sp9\" (UniqueName: \"kubernetes.io/projected/80c65b8e-8533-469e-bb66-50f25eaef7e9-kube-api-access-h5sp9\") pod \"isvc-xgboost-v2-predictor-7d5fcf856-nj9pj\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") " pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:24:40.246629 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.246603    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:24:40.667376 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:40.667344    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"]
Oct 30 19:24:40.674035 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:24:40.674011    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c65b8e_8533_469e_bb66_50f25eaef7e9.slice/crio-9706deaaa586ae8b7a6652a1ff304ae73112d9870b5233b8428c95b657d43c37 WatchSource:0}: Error finding container 9706deaaa586ae8b7a6652a1ff304ae73112d9870b5233b8428c95b657d43c37: Status 404 returned error can't find the container with id 9706deaaa586ae8b7a6652a1ff304ae73112d9870b5233b8428c95b657d43c37
Oct 30 19:24:41.214055 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:41.214020    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" event={"ID":"80c65b8e-8533-469e-bb66-50f25eaef7e9","Type":"ContainerStarted","Data":"2fd48f2622710aa56f26c1bba465d3ee9ddc38d92678ce7be74e54fe010c01aa"}
Oct 30 19:24:41.214578 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:41.214076    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" event={"ID":"80c65b8e-8533-469e-bb66-50f25eaef7e9","Type":"ContainerStarted","Data":"9706deaaa586ae8b7a6652a1ff304ae73112d9870b5233b8428c95b657d43c37"}
Oct 30 19:24:45.231652 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:45.231619    4511 generic.go:356] "Generic (PLEG): container finished" podID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerID="2fd48f2622710aa56f26c1bba465d3ee9ddc38d92678ce7be74e54fe010c01aa" exitCode=0
Oct 30 19:24:45.232039 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:45.231698    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" event={"ID":"80c65b8e-8533-469e-bb66-50f25eaef7e9","Type":"ContainerDied","Data":"2fd48f2622710aa56f26c1bba465d3ee9ddc38d92678ce7be74e54fe010c01aa"}
Oct 30 19:24:45.233025 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:45.233011    4511 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:24:46.167989 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:46.167953    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 30 19:24:47.172237 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:47.172191    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 30 19:24:56.167357 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:56.167306    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 30 19:24:57.172568 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:24:57.172523    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 30 19:25:05.328019 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:05.327983    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" event={"ID":"80c65b8e-8533-469e-bb66-50f25eaef7e9","Type":"ContainerStarted","Data":"a646d1a8d404986e5a3a2999bb658173ccdd6323ff811de1e0574a96634034ae"}
Oct 30 19:25:05.328399 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:05.328072    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" event={"ID":"80c65b8e-8533-469e-bb66-50f25eaef7e9","Type":"ContainerStarted","Data":"4586ebb6ad08157133e8005c055f2fdd38139184b89434c0a7515f57fd6c962e"}
Oct 30 19:25:05.328399 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:05.328282    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:25:05.492524 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:05.492471    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" podStartSLOduration=3.345617158 podStartE2EDuration="26.492456856s" totalImagesPullingTime="19.24172752s" totalInitContainerRuntime="3.905112178s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:05.491057231 +0000 UTC m=+2800.008742154" watchObservedRunningTime="2026-10-30 19:25:05.492456856 +0000 UTC m=+2800.010141756"
Oct 30 19:25:06.167426 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:06.167389    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 30 19:25:06.332591 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:06.332556    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:25:06.333856 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:06.333829    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 30 19:25:07.172729 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:07.172684    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 30 19:25:07.336675 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:07.336644    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 30 19:25:10.156261 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:25:10.156224    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:25:12.341560 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:12.341526    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:25:12.342110 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:12.342083    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 30 19:25:16.167462 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:16.167419    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 30 19:25:17.172924 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:17.172882    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 30 19:25:22.342882 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:22.342841    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 30 19:25:26.167305 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:26.167267    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 30 19:25:27.172621 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:27.172575    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 30 19:25:32.342876 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:32.342837    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 30 19:25:36.167331 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:36.167232    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 30 19:25:37.172770 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:37.172731    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 30 19:25:40.156523 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:25:40.156490    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:25:42.342809 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:42.342772    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 30 19:25:46.167222 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:46.167182    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 30 19:25:47.172368 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:47.172326    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Oct 30 19:25:52.342940 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:52.342904    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 30 19:25:56.168063 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:56.168030    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:25:57.173504 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:57.173467    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:25:57.546931 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:57.546898    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"]
Oct 30 19:25:57.547207 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:57.547184    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kserve-container" containerID="cri-o://fd84b53f1547e1b0c4e69e0b10a099d853247454716e406899dbcc938ab3c008" gracePeriod=30
Oct 30 19:25:57.547328 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:57.547197    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kube-rbac-proxy" containerID="cri-o://68563aec61a0038b0e5e495c5a430b0276f4129ea9d64e45f101834d548a0190" gracePeriod=30
Oct 30 19:25:58.542867 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:58.542835    4511 generic.go:356] "Generic (PLEG): container finished" podID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerID="68563aec61a0038b0e5e495c5a430b0276f4129ea9d64e45f101834d548a0190" exitCode=0
Oct 30 19:25:58.543341 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:25:58.542908    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" event={"ID":"77df7a1b-02b2-48df-809f-e3ca34d79ef1","Type":"ContainerDied","Data":"68563aec61a0038b0e5e495c5a430b0276f4129ea9d64e45f101834d548a0190"}
Oct 30 19:26:02.188528 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.188503    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:26:02.298787 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.298756    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kube-api-access-tkp8k\" (UniqueName: \"kubernetes.io/projected/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kube-api-access-tkp8k\") pod \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") "
Oct 30 19:26:02.298958 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.298800    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77df7a1b-02b2-48df-809f-e3ca34d79ef1-proxy-tls\" (UniqueName: \"kubernetes.io/secret/77df7a1b-02b2-48df-809f-e3ca34d79ef1-proxy-tls\") pod \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") "
Oct 30 19:26:02.298958 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.298823    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kserve-provision-location\") pod \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") "
Oct 30 19:26:02.298958 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.298855    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kube-rbac-proxy-sar-config\") pod \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") "
Oct 30 19:26:02.298958 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.298876    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77df7a1b-02b2-48df-809f-e3ca34d79ef1-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77df7a1b-02b2-48df-809f-e3ca34d79ef1-cabundle-cert\") pod \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\" (UID: \"77df7a1b-02b2-48df-809f-e3ca34d79ef1\") "
Oct 30 19:26:02.299237 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.299207    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kserve-provision-location" pod "77df7a1b-02b2-48df-809f-e3ca34d79ef1" (UID: "77df7a1b-02b2-48df-809f-e3ca34d79ef1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:02.299319 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.299294    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kube-rbac-proxy-sar-config" pod "77df7a1b-02b2-48df-809f-e3ca34d79ef1" (UID: "77df7a1b-02b2-48df-809f-e3ca34d79ef1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:02.299367 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.299305    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77df7a1b-02b2-48df-809f-e3ca34d79ef1-cabundle-cert" pod "77df7a1b-02b2-48df-809f-e3ca34d79ef1" (UID: "77df7a1b-02b2-48df-809f-e3ca34d79ef1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:02.301151 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.301115    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kube-api-access-tkp8k" pod "77df7a1b-02b2-48df-809f-e3ca34d79ef1" (UID: "77df7a1b-02b2-48df-809f-e3ca34d79ef1"). InnerVolumeSpecName "kube-api-access-tkp8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:02.301228 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.301167    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77df7a1b-02b2-48df-809f-e3ca34d79ef1-proxy-tls" pod "77df7a1b-02b2-48df-809f-e3ca34d79ef1" (UID: "77df7a1b-02b2-48df-809f-e3ca34d79ef1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:02.342656 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.342622    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 30 19:26:02.400284 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.400252    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkp8k\" (UniqueName: \"kubernetes.io/projected/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kube-api-access-tkp8k\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:02.400284 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.400280    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77df7a1b-02b2-48df-809f-e3ca34d79ef1-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:02.400462 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.400294    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:02.400462 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.400307    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77df7a1b-02b2-48df-809f-e3ca34d79ef1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:02.400462 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.400320    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77df7a1b-02b2-48df-809f-e3ca34d79ef1-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:02.562327 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.562244    4511 generic.go:356] "Generic (PLEG): container finished" podID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerID="fd84b53f1547e1b0c4e69e0b10a099d853247454716e406899dbcc938ab3c008" exitCode=0
Oct 30 19:26:02.562327 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.562320    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"
Oct 30 19:26:02.562525 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.562322    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" event={"ID":"77df7a1b-02b2-48df-809f-e3ca34d79ef1","Type":"ContainerDied","Data":"fd84b53f1547e1b0c4e69e0b10a099d853247454716e406899dbcc938ab3c008"}
Oct 30 19:26:02.562525 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.562472    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" event={"ID":"77df7a1b-02b2-48df-809f-e3ca34d79ef1","Type":"ContainerDied","Data":"c8248cb74cbde688daf15100a927c46d99595f8083f61b592df770ff88437f42"}
Oct 30 19:26:02.562525 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.562509    4511 scope.go:122] "RemoveContainer" containerID="68563aec61a0038b0e5e495c5a430b0276f4129ea9d64e45f101834d548a0190"
Oct 30 19:26:02.570973 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.570953    4511 scope.go:122] "RemoveContainer" containerID="fd84b53f1547e1b0c4e69e0b10a099d853247454716e406899dbcc938ab3c008"
Oct 30 19:26:02.578294 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.578277    4511 scope.go:122] "RemoveContainer" containerID="b9b339d2102ae7eee0b3f5711db6741da3a70f4b8e91d0047b0b0fce34a13eec"
Oct 30 19:26:02.585276 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.585259    4511 scope.go:122] "RemoveContainer" containerID="68563aec61a0038b0e5e495c5a430b0276f4129ea9d64e45f101834d548a0190"
Oct 30 19:26:02.585524 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:26:02.585506    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68563aec61a0038b0e5e495c5a430b0276f4129ea9d64e45f101834d548a0190\": container with ID starting with 68563aec61a0038b0e5e495c5a430b0276f4129ea9d64e45f101834d548a0190 not found: ID does not exist" containerID="68563aec61a0038b0e5e495c5a430b0276f4129ea9d64e45f101834d548a0190"
Oct 30 19:26:02.585603 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.585539    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68563aec61a0038b0e5e495c5a430b0276f4129ea9d64e45f101834d548a0190"} err="failed to get container status \"68563aec61a0038b0e5e495c5a430b0276f4129ea9d64e45f101834d548a0190\": rpc error: code = NotFound desc = could not find container \"68563aec61a0038b0e5e495c5a430b0276f4129ea9d64e45f101834d548a0190\": container with ID starting with 68563aec61a0038b0e5e495c5a430b0276f4129ea9d64e45f101834d548a0190 not found: ID does not exist"
Oct 30 19:26:02.585603 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.585565    4511 scope.go:122] "RemoveContainer" containerID="fd84b53f1547e1b0c4e69e0b10a099d853247454716e406899dbcc938ab3c008"
Oct 30 19:26:02.585816 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:26:02.585799    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd84b53f1547e1b0c4e69e0b10a099d853247454716e406899dbcc938ab3c008\": container with ID starting with fd84b53f1547e1b0c4e69e0b10a099d853247454716e406899dbcc938ab3c008 not found: ID does not exist" containerID="fd84b53f1547e1b0c4e69e0b10a099d853247454716e406899dbcc938ab3c008"
Oct 30 19:26:02.585854 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.585822    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd84b53f1547e1b0c4e69e0b10a099d853247454716e406899dbcc938ab3c008"} err="failed to get container status \"fd84b53f1547e1b0c4e69e0b10a099d853247454716e406899dbcc938ab3c008\": rpc error: code = NotFound desc = could not find container \"fd84b53f1547e1b0c4e69e0b10a099d853247454716e406899dbcc938ab3c008\": container with ID starting with fd84b53f1547e1b0c4e69e0b10a099d853247454716e406899dbcc938ab3c008 not found: ID does not exist"
Oct 30 19:26:02.585854 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.585838    4511 scope.go:122] "RemoveContainer" containerID="b9b339d2102ae7eee0b3f5711db6741da3a70f4b8e91d0047b0b0fce34a13eec"
Oct 30 19:26:02.586069 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:26:02.586049    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b339d2102ae7eee0b3f5711db6741da3a70f4b8e91d0047b0b0fce34a13eec\": container with ID starting with b9b339d2102ae7eee0b3f5711db6741da3a70f4b8e91d0047b0b0fce34a13eec not found: ID does not exist" containerID="b9b339d2102ae7eee0b3f5711db6741da3a70f4b8e91d0047b0b0fce34a13eec"
Oct 30 19:26:02.586111 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.586076    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b339d2102ae7eee0b3f5711db6741da3a70f4b8e91d0047b0b0fce34a13eec"} err="failed to get container status \"b9b339d2102ae7eee0b3f5711db6741da3a70f4b8e91d0047b0b0fce34a13eec\": rpc error: code = NotFound desc = could not find container \"b9b339d2102ae7eee0b3f5711db6741da3a70f4b8e91d0047b0b0fce34a13eec\": container with ID starting with b9b339d2102ae7eee0b3f5711db6741da3a70f4b8e91d0047b0b0fce34a13eec not found: ID does not exist"
Oct 30 19:26:02.720250 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.720215    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"]
Oct 30 19:26:02.724268 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:02.724239    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv"]
Oct 30 19:26:03.167570 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:03.167531    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-46lcv" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": context deadline exceeded"
Oct 30 19:26:03.655976 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:03.655943    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" path="/var/lib/kubelet/pods/77df7a1b-02b2-48df-809f-e3ca34d79ef1/volumes"
Oct 30 19:26:04.468719 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:04.468685    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"]
Oct 30 19:26:04.468954 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:04.468926    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kserve-container" containerID="cri-o://c0430983628d03d9649debde7378da0343634d7f5fb29defd9c752363f4e118a" gracePeriod=30
Oct 30 19:26:04.468999 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:04.468972    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kube-rbac-proxy" containerID="cri-o://c4aaafe9298b8ea906250b654e461bf6b4489de98a62997e482e5c153e5ccb8a" gracePeriod=30
Oct 30 19:26:04.843319 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:04.843285    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"]
Oct 30 19:26:04.843706 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:04.843604    4511 cpu_manager.go:414] "containerMap: removing container" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="storage-initializer"
Oct 30 19:26:04.843706 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:04.843614    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="storage-initializer"
Oct 30 19:26:04.843706 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:04.843626    4511 cpu_manager.go:414] "containerMap: removing container" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kserve-container"
Oct 30 19:26:04.843706 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:04.843631    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kserve-container"
Oct 30 19:26:04.843706 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:04.843640    4511 cpu_manager.go:414] "containerMap: removing container" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kube-rbac-proxy"
Oct 30 19:26:04.843706 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:04.843645    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kube-rbac-proxy"
Oct 30 19:26:04.843706 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:04.843709    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kserve-container"
Oct 30 19:26:04.843921 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:04.843715    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="77df7a1b-02b2-48df-809f-e3ca34d79ef1" containerName="kube-rbac-proxy"
Oct 30 19:26:04.845818 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:04.845798    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:04.895991 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:04.895960    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-rollback-predictor-serving-cert\""
Oct 30 19:26:04.907364 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:04.907332    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"]
Oct 30 19:26:05.022760 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.022724    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bdd45c6-83a8-4b00-b716-96a72fceda14-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-dz5ls\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:05.022934 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.022832    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fffgf\" (UniqueName: \"kubernetes.io/projected/9bdd45c6-83a8-4b00-b716-96a72fceda14-kube-api-access-fffgf\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-dz5ls\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:05.022934 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.022864    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bdd45c6-83a8-4b00-b716-96a72fceda14-proxy-tls\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-dz5ls\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:05.022934 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.022883    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bdd45c6-83a8-4b00-b716-96a72fceda14-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-dz5ls\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:05.023076 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.022952    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bdd45c6-83a8-4b00-b716-96a72fceda14-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-dz5ls\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:05.124214 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.124122    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bdd45c6-83a8-4b00-b716-96a72fceda14-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-dz5ls\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:05.124359 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.124214    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fffgf\" (UniqueName: \"kubernetes.io/projected/9bdd45c6-83a8-4b00-b716-96a72fceda14-kube-api-access-fffgf\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-dz5ls\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:05.124359 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.124239    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bdd45c6-83a8-4b00-b716-96a72fceda14-proxy-tls\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-dz5ls\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:05.124359 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.124260    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bdd45c6-83a8-4b00-b716-96a72fceda14-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-dz5ls\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:05.124359 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.124283    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bdd45c6-83a8-4b00-b716-96a72fceda14-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-dz5ls\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:05.124574 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.124510    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bdd45c6-83a8-4b00-b716-96a72fceda14-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-dz5ls\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:05.124928 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.124907    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bdd45c6-83a8-4b00-b716-96a72fceda14-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-dz5ls\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:05.125033 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.125011    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bdd45c6-83a8-4b00-b716-96a72fceda14-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-dz5ls\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:05.126778 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.126760    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bdd45c6-83a8-4b00-b716-96a72fceda14-proxy-tls\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-dz5ls\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:05.183276 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.183249    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fffgf\" (UniqueName: \"kubernetes.io/projected/9bdd45c6-83a8-4b00-b716-96a72fceda14-kube-api-access-fffgf\") pod \"isvc-canary-rollback-predictor-7989d9dc9c-dz5ls\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") " pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:05.459466 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.459387    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:05.577972 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.577941    4511 generic.go:356] "Generic (PLEG): container finished" podID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerID="c4aaafe9298b8ea906250b654e461bf6b4489de98a62997e482e5c153e5ccb8a" exitCode=0
Oct 30 19:26:05.578109 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.578012    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" event={"ID":"eb826157-08bd-4622-a2c6-b6ba1240601a","Type":"ContainerDied","Data":"c4aaafe9298b8ea906250b654e461bf6b4489de98a62997e482e5c153e5ccb8a"}
Oct 30 19:26:05.881360 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:05.881323    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"]
Oct 30 19:26:05.888926 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:26:05.888891    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bdd45c6_83a8_4b00_b716_96a72fceda14.slice/crio-a1f9978fe55df42fcc95aee1d3981dd8a77311226441d4173557552368211908 WatchSource:0}: Error finding container a1f9978fe55df42fcc95aee1d3981dd8a77311226441d4173557552368211908: Status 404 returned error can't find the container with id a1f9978fe55df42fcc95aee1d3981dd8a77311226441d4173557552368211908
Oct 30 19:26:06.162755 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:06.162660    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Oct 30 19:26:06.168001 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:06.167969    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Oct 30 19:26:06.583252 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:06.583220    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" event={"ID":"9bdd45c6-83a8-4b00-b716-96a72fceda14","Type":"ContainerStarted","Data":"407ec184336f915494a451237550f3de9408f6e0dba83ec3414950910f9f64b4"}
Oct 30 19:26:06.583454 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:06.583280    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" event={"ID":"9bdd45c6-83a8-4b00-b716-96a72fceda14","Type":"ContainerStarted","Data":"a1f9978fe55df42fcc95aee1d3981dd8a77311226441d4173557552368211908"}
Oct 30 19:26:09.301702 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.301678    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:26:09.358419 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.358396    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb826157-08bd-4622-a2c6-b6ba1240601a-proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb826157-08bd-4622-a2c6-b6ba1240601a-proxy-tls\") pod \"eb826157-08bd-4622-a2c6-b6ba1240601a\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") "
Oct 30 19:26:09.358552 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.358433    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb826157-08bd-4622-a2c6-b6ba1240601a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb826157-08bd-4622-a2c6-b6ba1240601a-kube-rbac-proxy-sar-config\") pod \"eb826157-08bd-4622-a2c6-b6ba1240601a\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") "
Oct 30 19:26:09.358552 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.358464    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb826157-08bd-4622-a2c6-b6ba1240601a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb826157-08bd-4622-a2c6-b6ba1240601a-cabundle-cert\") pod \"eb826157-08bd-4622-a2c6-b6ba1240601a\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") "
Oct 30 19:26:09.358552 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.358505    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb826157-08bd-4622-a2c6-b6ba1240601a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb826157-08bd-4622-a2c6-b6ba1240601a-kserve-provision-location\") pod \"eb826157-08bd-4622-a2c6-b6ba1240601a\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") "
Oct 30 19:26:09.358552 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.358535    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb826157-08bd-4622-a2c6-b6ba1240601a-kube-api-access-vwlrk\" (UniqueName: \"kubernetes.io/projected/eb826157-08bd-4622-a2c6-b6ba1240601a-kube-api-access-vwlrk\") pod \"eb826157-08bd-4622-a2c6-b6ba1240601a\" (UID: \"eb826157-08bd-4622-a2c6-b6ba1240601a\") "
Oct 30 19:26:09.358836 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.358817    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb826157-08bd-4622-a2c6-b6ba1240601a-cabundle-cert" pod "eb826157-08bd-4622-a2c6-b6ba1240601a" (UID: "eb826157-08bd-4622-a2c6-b6ba1240601a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:09.358836 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.358821    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb826157-08bd-4622-a2c6-b6ba1240601a-kube-rbac-proxy-sar-config" pod "eb826157-08bd-4622-a2c6-b6ba1240601a" (UID: "eb826157-08bd-4622-a2c6-b6ba1240601a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:09.358920 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.358862    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb826157-08bd-4622-a2c6-b6ba1240601a-kserve-provision-location" pod "eb826157-08bd-4622-a2c6-b6ba1240601a" (UID: "eb826157-08bd-4622-a2c6-b6ba1240601a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:09.360500 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.360477    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb826157-08bd-4622-a2c6-b6ba1240601a-proxy-tls" pod "eb826157-08bd-4622-a2c6-b6ba1240601a" (UID: "eb826157-08bd-4622-a2c6-b6ba1240601a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:09.360605 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.360585    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb826157-08bd-4622-a2c6-b6ba1240601a-kube-api-access-vwlrk" pod "eb826157-08bd-4622-a2c6-b6ba1240601a" (UID: "eb826157-08bd-4622-a2c6-b6ba1240601a"). InnerVolumeSpecName "kube-api-access-vwlrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:09.459283 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.459217    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb826157-08bd-4622-a2c6-b6ba1240601a-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:09.459283 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.459243    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb826157-08bd-4622-a2c6-b6ba1240601a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:09.459283 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.459253    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb826157-08bd-4622-a2c6-b6ba1240601a-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:09.459283 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.459265    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb826157-08bd-4622-a2c6-b6ba1240601a-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:09.459283 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.459274    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwlrk\" (UniqueName: \"kubernetes.io/projected/eb826157-08bd-4622-a2c6-b6ba1240601a-kube-api-access-vwlrk\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:09.598017 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.597984    4511 generic.go:356] "Generic (PLEG): container finished" podID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerID="c0430983628d03d9649debde7378da0343634d7f5fb29defd9c752363f4e118a" exitCode=0
Oct 30 19:26:09.598193 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.598035    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" event={"ID":"eb826157-08bd-4622-a2c6-b6ba1240601a","Type":"ContainerDied","Data":"c0430983628d03d9649debde7378da0343634d7f5fb29defd9c752363f4e118a"}
Oct 30 19:26:09.598193 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.598041    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"
Oct 30 19:26:09.598193 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.598086    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx" event={"ID":"eb826157-08bd-4622-a2c6-b6ba1240601a","Type":"ContainerDied","Data":"04a1193a0706f0b99f47be241ff284891051da47bbde872d91ee06e210391a49"}
Oct 30 19:26:09.598193 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.598120    4511 scope.go:122] "RemoveContainer" containerID="c4aaafe9298b8ea906250b654e461bf6b4489de98a62997e482e5c153e5ccb8a"
Oct 30 19:26:09.606732 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.606713    4511 scope.go:122] "RemoveContainer" containerID="c0430983628d03d9649debde7378da0343634d7f5fb29defd9c752363f4e118a"
Oct 30 19:26:09.613872 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.613856    4511 scope.go:122] "RemoveContainer" containerID="8043aecd1582d59595beb0cec97e56e2c3ac61329e92e769c1f9fd02e3e42efd"
Oct 30 19:26:09.620970 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.620956    4511 scope.go:122] "RemoveContainer" containerID="c4aaafe9298b8ea906250b654e461bf6b4489de98a62997e482e5c153e5ccb8a"
Oct 30 19:26:09.621230 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:26:09.621212    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4aaafe9298b8ea906250b654e461bf6b4489de98a62997e482e5c153e5ccb8a\": container with ID starting with c4aaafe9298b8ea906250b654e461bf6b4489de98a62997e482e5c153e5ccb8a not found: ID does not exist" containerID="c4aaafe9298b8ea906250b654e461bf6b4489de98a62997e482e5c153e5ccb8a"
Oct 30 19:26:09.621292 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.621239    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4aaafe9298b8ea906250b654e461bf6b4489de98a62997e482e5c153e5ccb8a"} err="failed to get container status \"c4aaafe9298b8ea906250b654e461bf6b4489de98a62997e482e5c153e5ccb8a\": rpc error: code = NotFound desc = could not find container \"c4aaafe9298b8ea906250b654e461bf6b4489de98a62997e482e5c153e5ccb8a\": container with ID starting with c4aaafe9298b8ea906250b654e461bf6b4489de98a62997e482e5c153e5ccb8a not found: ID does not exist"
Oct 30 19:26:09.621292 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.621255    4511 scope.go:122] "RemoveContainer" containerID="c0430983628d03d9649debde7378da0343634d7f5fb29defd9c752363f4e118a"
Oct 30 19:26:09.621468 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:26:09.621453    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0430983628d03d9649debde7378da0343634d7f5fb29defd9c752363f4e118a\": container with ID starting with c0430983628d03d9649debde7378da0343634d7f5fb29defd9c752363f4e118a not found: ID does not exist" containerID="c0430983628d03d9649debde7378da0343634d7f5fb29defd9c752363f4e118a"
Oct 30 19:26:09.621511 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.621472    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0430983628d03d9649debde7378da0343634d7f5fb29defd9c752363f4e118a"} err="failed to get container status \"c0430983628d03d9649debde7378da0343634d7f5fb29defd9c752363f4e118a\": rpc error: code = NotFound desc = could not find container \"c0430983628d03d9649debde7378da0343634d7f5fb29defd9c752363f4e118a\": container with ID starting with c0430983628d03d9649debde7378da0343634d7f5fb29defd9c752363f4e118a not found: ID does not exist"
Oct 30 19:26:09.621511 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.621485    4511 scope.go:122] "RemoveContainer" containerID="8043aecd1582d59595beb0cec97e56e2c3ac61329e92e769c1f9fd02e3e42efd"
Oct 30 19:26:09.621700 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:26:09.621681    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8043aecd1582d59595beb0cec97e56e2c3ac61329e92e769c1f9fd02e3e42efd\": container with ID starting with 8043aecd1582d59595beb0cec97e56e2c3ac61329e92e769c1f9fd02e3e42efd not found: ID does not exist" containerID="8043aecd1582d59595beb0cec97e56e2c3ac61329e92e769c1f9fd02e3e42efd"
Oct 30 19:26:09.621753 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.621712    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8043aecd1582d59595beb0cec97e56e2c3ac61329e92e769c1f9fd02e3e42efd"} err="failed to get container status \"8043aecd1582d59595beb0cec97e56e2c3ac61329e92e769c1f9fd02e3e42efd\": rpc error: code = NotFound desc = could not find container \"8043aecd1582d59595beb0cec97e56e2c3ac61329e92e769c1f9fd02e3e42efd\": container with ID starting with 8043aecd1582d59595beb0cec97e56e2c3ac61329e92e769c1f9fd02e3e42efd not found: ID does not exist"
Oct 30 19:26:09.756986 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.756911    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"]
Oct 30 19:26:09.761205 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:09.761180    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-v2-predictor-795c78bfcd-9hwqx"]
Oct 30 19:26:10.156261 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:26:10.156228    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:26:10.604853 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:10.604816    4511 generic.go:356] "Generic (PLEG): container finished" podID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerID="407ec184336f915494a451237550f3de9408f6e0dba83ec3414950910f9f64b4" exitCode=0
Oct 30 19:26:10.605358 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:10.604882    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" event={"ID":"9bdd45c6-83a8-4b00-b716-96a72fceda14","Type":"ContainerDied","Data":"407ec184336f915494a451237550f3de9408f6e0dba83ec3414950910f9f64b4"}
Oct 30 19:26:11.610855 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:11.610818    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" event={"ID":"9bdd45c6-83a8-4b00-b716-96a72fceda14","Type":"ContainerStarted","Data":"9570d148e2b429f55481e842e3f4a7e8889f29f75d3708cea4b260105b68725f"}
Oct 30 19:26:11.611263 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:11.610882    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" event={"ID":"9bdd45c6-83a8-4b00-b716-96a72fceda14","Type":"ContainerStarted","Data":"6054225f5480b97b1ccef53fb3349285304e0f50d92ed6700ae5a20b5b565f29"}
Oct 30 19:26:11.611263 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:11.611076    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:11.655542 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:11.655504    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" path="/var/lib/kubelet/pods/eb826157-08bd-4622-a2c6-b6ba1240601a/volumes"
Oct 30 19:26:11.727623 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:11.727566    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" podStartSLOduration=3.8498680690000002 podStartE2EDuration="7.727551007s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.877682938s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:11.725766976 +0000 UTC m=+2866.243451887" watchObservedRunningTime="2026-10-30 19:26:11.727551007 +0000 UTC m=+2866.245235908"
Oct 30 19:26:12.343474 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:12.343446    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:26:12.614671 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:12.614586    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:12.615933 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:12.615907    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 30 19:26:13.618894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:13.618842    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 30 19:26:18.623365 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:18.623339    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:26:18.623863 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:18.623838    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 30 19:26:20.744216 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:20.744185    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"]
Oct 30 19:26:20.744598 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:20.744438    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kserve-container" containerID="cri-o://4586ebb6ad08157133e8005c055f2fdd38139184b89434c0a7515f57fd6c962e" gracePeriod=30
Oct 30 19:26:20.744598 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:20.744472    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kube-rbac-proxy" containerID="cri-o://a646d1a8d404986e5a3a2999bb658173ccdd6323ff811de1e0574a96634034ae" gracePeriod=30
Oct 30 19:26:21.653561 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:21.653528    4511 generic.go:356] "Generic (PLEG): container finished" podID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerID="a646d1a8d404986e5a3a2999bb658173ccdd6323ff811de1e0574a96634034ae" exitCode=0
Oct 30 19:26:21.655716 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:21.655692    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" event={"ID":"80c65b8e-8533-469e-bb66-50f25eaef7e9","Type":"ContainerDied","Data":"a646d1a8d404986e5a3a2999bb658173ccdd6323ff811de1e0574a96634034ae"}
Oct 30 19:26:22.337655 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:22.337608    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Oct 30 19:26:22.342934 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:22.342911    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Oct 30 19:26:24.668573 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.668543    4511 generic.go:356] "Generic (PLEG): container finished" podID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerID="4586ebb6ad08157133e8005c055f2fdd38139184b89434c0a7515f57fd6c962e" exitCode=0
Oct 30 19:26:24.668907 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.668612    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" event={"ID":"80c65b8e-8533-469e-bb66-50f25eaef7e9","Type":"ContainerDied","Data":"4586ebb6ad08157133e8005c055f2fdd38139184b89434c0a7515f57fd6c962e"}
Oct 30 19:26:24.683628 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.683607    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:26:24.772446 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.772379    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c65b8e-8533-469e-bb66-50f25eaef7e9-proxy-tls\" (UniqueName: \"kubernetes.io/secret/80c65b8e-8533-469e-bb66-50f25eaef7e9-proxy-tls\") pod \"80c65b8e-8533-469e-bb66-50f25eaef7e9\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") "
Oct 30 19:26:24.772446 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.772425    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c65b8e-8533-469e-bb66-50f25eaef7e9-kube-api-access-h5sp9\" (UniqueName: \"kubernetes.io/projected/80c65b8e-8533-469e-bb66-50f25eaef7e9-kube-api-access-h5sp9\") pod \"80c65b8e-8533-469e-bb66-50f25eaef7e9\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") "
Oct 30 19:26:24.772608 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.772450    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c65b8e-8533-469e-bb66-50f25eaef7e9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80c65b8e-8533-469e-bb66-50f25eaef7e9-kube-rbac-proxy-sar-config\") pod \"80c65b8e-8533-469e-bb66-50f25eaef7e9\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") "
Oct 30 19:26:24.772608 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.772549    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c65b8e-8533-469e-bb66-50f25eaef7e9-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80c65b8e-8533-469e-bb66-50f25eaef7e9-cabundle-cert\") pod \"80c65b8e-8533-469e-bb66-50f25eaef7e9\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") "
Oct 30 19:26:24.772695 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.772621    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c65b8e-8533-469e-bb66-50f25eaef7e9-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80c65b8e-8533-469e-bb66-50f25eaef7e9-kserve-provision-location\") pod \"80c65b8e-8533-469e-bb66-50f25eaef7e9\" (UID: \"80c65b8e-8533-469e-bb66-50f25eaef7e9\") "
Oct 30 19:26:24.772822 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.772793    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c65b8e-8533-469e-bb66-50f25eaef7e9-kube-rbac-proxy-sar-config" pod "80c65b8e-8533-469e-bb66-50f25eaef7e9" (UID: "80c65b8e-8533-469e-bb66-50f25eaef7e9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:24.772936 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.772905    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c65b8e-8533-469e-bb66-50f25eaef7e9-cabundle-cert" pod "80c65b8e-8533-469e-bb66-50f25eaef7e9" (UID: "80c65b8e-8533-469e-bb66-50f25eaef7e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:24.772936 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.772901    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c65b8e-8533-469e-bb66-50f25eaef7e9-kserve-provision-location" pod "80c65b8e-8533-469e-bb66-50f25eaef7e9" (UID: "80c65b8e-8533-469e-bb66-50f25eaef7e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:24.775205 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.775180    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c65b8e-8533-469e-bb66-50f25eaef7e9-proxy-tls" pod "80c65b8e-8533-469e-bb66-50f25eaef7e9" (UID: "80c65b8e-8533-469e-bb66-50f25eaef7e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:24.775289 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.775207    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c65b8e-8533-469e-bb66-50f25eaef7e9-kube-api-access-h5sp9" pod "80c65b8e-8533-469e-bb66-50f25eaef7e9" (UID: "80c65b8e-8533-469e-bb66-50f25eaef7e9"). InnerVolumeSpecName "kube-api-access-h5sp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:24.873526 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.873499    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80c65b8e-8533-469e-bb66-50f25eaef7e9-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:24.873526 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.873521    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5sp9\" (UniqueName: \"kubernetes.io/projected/80c65b8e-8533-469e-bb66-50f25eaef7e9-kube-api-access-h5sp9\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:24.873526 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.873531    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80c65b8e-8533-469e-bb66-50f25eaef7e9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:24.873728 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.873543    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80c65b8e-8533-469e-bb66-50f25eaef7e9-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:24.873728 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:24.873551    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80c65b8e-8533-469e-bb66-50f25eaef7e9-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:25.673984 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:25.673958    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"
Oct 30 19:26:25.674349 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:25.673963    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj" event={"ID":"80c65b8e-8533-469e-bb66-50f25eaef7e9","Type":"ContainerDied","Data":"9706deaaa586ae8b7a6652a1ff304ae73112d9870b5233b8428c95b657d43c37"}
Oct 30 19:26:25.674349 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:25.674108    4511 scope.go:122] "RemoveContainer" containerID="a646d1a8d404986e5a3a2999bb658173ccdd6323ff811de1e0574a96634034ae"
Oct 30 19:26:25.682073 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:25.682044    4511 scope.go:122] "RemoveContainer" containerID="4586ebb6ad08157133e8005c055f2fdd38139184b89434c0a7515f57fd6c962e"
Oct 30 19:26:25.689211 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:25.689193    4511 scope.go:122] "RemoveContainer" containerID="2fd48f2622710aa56f26c1bba465d3ee9ddc38d92678ce7be74e54fe010c01aa"
Oct 30 19:26:25.880200 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:25.880165    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"]
Oct 30 19:26:25.887164 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:25.887140    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-xgboost-v2-predictor-7d5fcf856-nj9pj"]
Oct 30 19:26:27.655858 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:27.655824    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" path="/var/lib/kubelet/pods/80c65b8e-8533-469e-bb66-50f25eaef7e9/volumes"
Oct 30 19:26:28.623766 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:28.623721    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 30 19:26:30.078963 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.078921    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"]
Oct 30 19:26:30.079582 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.079379    4511 cpu_manager.go:414] "containerMap: removing container" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kube-rbac-proxy"
Oct 30 19:26:30.079582 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.079397    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kube-rbac-proxy"
Oct 30 19:26:30.079582 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.079425    4511 cpu_manager.go:414] "containerMap: removing container" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kube-rbac-proxy"
Oct 30 19:26:30.079582 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.079433    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kube-rbac-proxy"
Oct 30 19:26:30.079582 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.079451    4511 cpu_manager.go:414] "containerMap: removing container" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="storage-initializer"
Oct 30 19:26:30.079582 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.079458    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="storage-initializer"
Oct 30 19:26:30.079582 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.079470    4511 cpu_manager.go:414] "containerMap: removing container" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kserve-container"
Oct 30 19:26:30.079582 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.079478    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kserve-container"
Oct 30 19:26:30.079582 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.079494    4511 cpu_manager.go:414] "containerMap: removing container" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="storage-initializer"
Oct 30 19:26:30.079582 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.079502    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="storage-initializer"
Oct 30 19:26:30.079582 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.079511    4511 cpu_manager.go:414] "containerMap: removing container" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kserve-container"
Oct 30 19:26:30.079582 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.079519    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kserve-container"
Oct 30 19:26:30.080195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.079593    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kube-rbac-proxy"
Oct 30 19:26:30.080195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.079606    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kube-rbac-proxy"
Oct 30 19:26:30.080195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.079615    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb826157-08bd-4622-a2c6-b6ba1240601a" containerName="kserve-container"
Oct 30 19:26:30.080195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.079624    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c65b8e-8533-469e-bb66-50f25eaef7e9" containerName="kserve-container"
Oct 30 19:26:30.083477 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.083454    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:26:30.111685 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.111666    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9039b99-5a63-484f-ac10-1bb91c50a1cb-proxy-tls\") pod \"isvc-sklearn-s3-predictor-777575875d-sbr7p\" (UID: \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:26:30.111796 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.111692    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-777575875d-sbr7p\" (UID: \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:26:30.111796 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.111729    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w2mk\" (UniqueName: \"kubernetes.io/projected/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kube-api-access-6w2mk\") pod \"isvc-sklearn-s3-predictor-777575875d-sbr7p\" (UID: \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:26:30.111870 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.111801    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-777575875d-sbr7p\" (UID: \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:26:30.134966 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.134943    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Oct 30 19:26:30.134966 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.134960    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:26:30.182182 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.182161    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:26:30.182279 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.182210    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"storage-config\""
Oct 30 19:26:30.182279 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.182257    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Oct 30 19:26:30.182354 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.182310    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:26:30.185046 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.185028    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"]
Oct 30 19:26:30.213035 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.213012    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2mk\" (UniqueName: \"kubernetes.io/projected/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kube-api-access-6w2mk\") pod \"isvc-sklearn-s3-predictor-777575875d-sbr7p\" (UID: \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:26:30.213148 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.213041    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-777575875d-sbr7p\" (UID: \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:26:30.213148 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.213088    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9039b99-5a63-484f-ac10-1bb91c50a1cb-proxy-tls\") pod \"isvc-sklearn-s3-predictor-777575875d-sbr7p\" (UID: \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:26:30.213148 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.213106    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-777575875d-sbr7p\" (UID: \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:26:30.213536 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.213515    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-777575875d-sbr7p\" (UID: \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:26:30.213722 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.213707    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-777575875d-sbr7p\" (UID: \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:26:30.215636 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.215617    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9039b99-5a63-484f-ac10-1bb91c50a1cb-proxy-tls\") pod \"isvc-sklearn-s3-predictor-777575875d-sbr7p\" (UID: \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:26:30.272154 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.272114    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w2mk\" (UniqueName: \"kubernetes.io/projected/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kube-api-access-6w2mk\") pod \"isvc-sklearn-s3-predictor-777575875d-sbr7p\" (UID: \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:26:30.397146 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.397060    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:26:30.818235 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:30.818197    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"]
Oct 30 19:26:30.824812 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:26:30.824786    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9039b99_5a63_484f_ac10_1bb91c50a1cb.slice/crio-88590e2f420bd43fe8b523cdbdbdc7e0bc1d00082c2a3f45c1649d334cfb4754 WatchSource:0}: Error finding container 88590e2f420bd43fe8b523cdbdbdc7e0bc1d00082c2a3f45c1649d334cfb4754: Status 404 returned error can't find the container with id 88590e2f420bd43fe8b523cdbdbdc7e0bc1d00082c2a3f45c1649d334cfb4754
Oct 30 19:26:31.695858 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:31.695821    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" event={"ID":"b9039b99-5a63-484f-ac10-1bb91c50a1cb","Type":"ContainerStarted","Data":"5fca3ce6cf1e722cd4b784d5fbc4ecf869366929cd3f50b0291becda4aeaf659"}
Oct 30 19:26:31.696353 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:31.695877    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" event={"ID":"b9039b99-5a63-484f-ac10-1bb91c50a1cb","Type":"ContainerStarted","Data":"88590e2f420bd43fe8b523cdbdbdc7e0bc1d00082c2a3f45c1649d334cfb4754"}
Oct 30 19:26:32.701286 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:32.701243    4511 generic.go:356] "Generic (PLEG): container finished" podID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerID="5fca3ce6cf1e722cd4b784d5fbc4ecf869366929cd3f50b0291becda4aeaf659" exitCode=0
Oct 30 19:26:32.701799 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:32.701316    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" event={"ID":"b9039b99-5a63-484f-ac10-1bb91c50a1cb","Type":"ContainerDied","Data":"5fca3ce6cf1e722cd4b784d5fbc4ecf869366929cd3f50b0291becda4aeaf659"}
Oct 30 19:26:33.706880 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:33.706844    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" event={"ID":"b9039b99-5a63-484f-ac10-1bb91c50a1cb","Type":"ContainerStarted","Data":"9a5e26d58d47b51af69fdabb11efa135532fd6ddad35c8b334b7db6ed3efba80"}
Oct 30 19:26:33.707307 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:33.706904    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" event={"ID":"b9039b99-5a63-484f-ac10-1bb91c50a1cb","Type":"ContainerStarted","Data":"e603518190beb2ca25b21dcebe1eb62e4263d57ff325b4b2c8776e07af6f3979"}
Oct 30 19:26:33.707307 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:33.707079    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:26:33.707307 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:33.707235    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:26:33.708697 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:33.708670    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 30 19:26:33.824051 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:33.823993    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" podStartSLOduration=2.703132547 podStartE2EDuration="3.823979965s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.120847418s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:33.822832223 +0000 UTC m=+2888.340517125" watchObservedRunningTime="2026-10-30 19:26:33.823979965 +0000 UTC m=+2888.341664865"
Oct 30 19:26:34.710697 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:34.710662    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 30 19:26:38.624445 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:38.624410    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 30 19:26:39.715364 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:39.715333    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:26:39.715939 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:39.715911    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 30 19:26:40.155981 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:26:40.155948    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:26:48.623856 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:48.623802    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 30 19:26:49.715869 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:49.715828    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 30 19:26:58.623824 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:58.623780    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 30 19:26:59.716682 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:26:59.716638    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 30 19:27:08.624544 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:08.624497    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 30 19:27:09.715887 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:09.715851    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 30 19:27:10.156514 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:27:10.156482    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:27:18.623904 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:18.623856    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 30 19:27:19.715803 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:19.715760    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 30 19:27:28.625327 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:28.625296    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:27:29.716248 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:29.716215    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 30 19:27:35.032997 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:35.032963    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"]
Oct 30 19:27:35.033427 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:35.033198    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kserve-container" containerID="cri-o://6054225f5480b97b1ccef53fb3349285304e0f50d92ed6700ae5a20b5b565f29" gracePeriod=30
Oct 30 19:27:35.033427 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:35.033245    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kube-rbac-proxy" containerID="cri-o://9570d148e2b429f55481e842e3f4a7e8889f29f75d3708cea4b260105b68725f" gracePeriod=30
Oct 30 19:27:35.946864 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:35.946830    4511 generic.go:356] "Generic (PLEG): container finished" podID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerID="9570d148e2b429f55481e842e3f4a7e8889f29f75d3708cea4b260105b68725f" exitCode=0
Oct 30 19:27:35.947034 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:35.946903    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" event={"ID":"9bdd45c6-83a8-4b00-b716-96a72fceda14","Type":"ContainerDied","Data":"9570d148e2b429f55481e842e3f4a7e8889f29f75d3708cea4b260105b68725f"}
Oct 30 19:27:38.620002 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:38.619955    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Oct 30 19:27:38.624363 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:38.624329    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Oct 30 19:27:39.573639 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.573615    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:27:39.652086 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.652060    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bdd45c6-83a8-4b00-b716-96a72fceda14-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bdd45c6-83a8-4b00-b716-96a72fceda14-kube-rbac-proxy-sar-config\") pod \"9bdd45c6-83a8-4b00-b716-96a72fceda14\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") "
Oct 30 19:27:39.652506 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.652118    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bdd45c6-83a8-4b00-b716-96a72fceda14-kube-api-access-fffgf\" (UniqueName: \"kubernetes.io/projected/9bdd45c6-83a8-4b00-b716-96a72fceda14-kube-api-access-fffgf\") pod \"9bdd45c6-83a8-4b00-b716-96a72fceda14\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") "
Oct 30 19:27:39.652506 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.652196    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bdd45c6-83a8-4b00-b716-96a72fceda14-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bdd45c6-83a8-4b00-b716-96a72fceda14-kserve-provision-location\") pod \"9bdd45c6-83a8-4b00-b716-96a72fceda14\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") "
Oct 30 19:27:39.652506 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.652236    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bdd45c6-83a8-4b00-b716-96a72fceda14-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bdd45c6-83a8-4b00-b716-96a72fceda14-proxy-tls\") pod \"9bdd45c6-83a8-4b00-b716-96a72fceda14\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") "
Oct 30 19:27:39.652506 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.652282    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bdd45c6-83a8-4b00-b716-96a72fceda14-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bdd45c6-83a8-4b00-b716-96a72fceda14-cabundle-cert\") pod \"9bdd45c6-83a8-4b00-b716-96a72fceda14\" (UID: \"9bdd45c6-83a8-4b00-b716-96a72fceda14\") "
Oct 30 19:27:39.652506 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.652467    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bdd45c6-83a8-4b00-b716-96a72fceda14-kube-rbac-proxy-sar-config" pod "9bdd45c6-83a8-4b00-b716-96a72fceda14" (UID: "9bdd45c6-83a8-4b00-b716-96a72fceda14"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:39.652771 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.652513    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bdd45c6-83a8-4b00-b716-96a72fceda14-kserve-provision-location" pod "9bdd45c6-83a8-4b00-b716-96a72fceda14" (UID: "9bdd45c6-83a8-4b00-b716-96a72fceda14"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:39.652771 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.652685    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bdd45c6-83a8-4b00-b716-96a72fceda14-cabundle-cert" pod "9bdd45c6-83a8-4b00-b716-96a72fceda14" (UID: "9bdd45c6-83a8-4b00-b716-96a72fceda14"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:39.654436 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.654411    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bdd45c6-83a8-4b00-b716-96a72fceda14-proxy-tls" pod "9bdd45c6-83a8-4b00-b716-96a72fceda14" (UID: "9bdd45c6-83a8-4b00-b716-96a72fceda14"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:39.654598 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.654582    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bdd45c6-83a8-4b00-b716-96a72fceda14-kube-api-access-fffgf" pod "9bdd45c6-83a8-4b00-b716-96a72fceda14" (UID: "9bdd45c6-83a8-4b00-b716-96a72fceda14"). InnerVolumeSpecName "kube-api-access-fffgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:39.716566 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.716511    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Oct 30 19:27:39.753692 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.753663    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bdd45c6-83a8-4b00-b716-96a72fceda14-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:39.753809 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.753697    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bdd45c6-83a8-4b00-b716-96a72fceda14-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:39.753809 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.753714    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fffgf\" (UniqueName: \"kubernetes.io/projected/9bdd45c6-83a8-4b00-b716-96a72fceda14-kube-api-access-fffgf\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:39.753809 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.753728    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bdd45c6-83a8-4b00-b716-96a72fceda14-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:39.753809 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.753743    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bdd45c6-83a8-4b00-b716-96a72fceda14-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:39.964366 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.964335    4511 generic.go:356] "Generic (PLEG): container finished" podID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerID="6054225f5480b97b1ccef53fb3349285304e0f50d92ed6700ae5a20b5b565f29" exitCode=0
Oct 30 19:27:39.964501 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.964413    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"
Oct 30 19:27:39.964501 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.964432    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" event={"ID":"9bdd45c6-83a8-4b00-b716-96a72fceda14","Type":"ContainerDied","Data":"6054225f5480b97b1ccef53fb3349285304e0f50d92ed6700ae5a20b5b565f29"}
Oct 30 19:27:39.964501 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.964482    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls" event={"ID":"9bdd45c6-83a8-4b00-b716-96a72fceda14","Type":"ContainerDied","Data":"a1f9978fe55df42fcc95aee1d3981dd8a77311226441d4173557552368211908"}
Oct 30 19:27:39.964613 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.964508    4511 scope.go:122] "RemoveContainer" containerID="9570d148e2b429f55481e842e3f4a7e8889f29f75d3708cea4b260105b68725f"
Oct 30 19:27:39.972440 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.972421    4511 scope.go:122] "RemoveContainer" containerID="6054225f5480b97b1ccef53fb3349285304e0f50d92ed6700ae5a20b5b565f29"
Oct 30 19:27:39.979340 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.979316    4511 scope.go:122] "RemoveContainer" containerID="407ec184336f915494a451237550f3de9408f6e0dba83ec3414950910f9f64b4"
Oct 30 19:27:39.986247 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.986231    4511 scope.go:122] "RemoveContainer" containerID="9570d148e2b429f55481e842e3f4a7e8889f29f75d3708cea4b260105b68725f"
Oct 30 19:27:39.986518 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:27:39.986502    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9570d148e2b429f55481e842e3f4a7e8889f29f75d3708cea4b260105b68725f\": container with ID starting with 9570d148e2b429f55481e842e3f4a7e8889f29f75d3708cea4b260105b68725f not found: ID does not exist" containerID="9570d148e2b429f55481e842e3f4a7e8889f29f75d3708cea4b260105b68725f"
Oct 30 19:27:39.986600 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.986531    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9570d148e2b429f55481e842e3f4a7e8889f29f75d3708cea4b260105b68725f"} err="failed to get container status \"9570d148e2b429f55481e842e3f4a7e8889f29f75d3708cea4b260105b68725f\": rpc error: code = NotFound desc = could not find container \"9570d148e2b429f55481e842e3f4a7e8889f29f75d3708cea4b260105b68725f\": container with ID starting with 9570d148e2b429f55481e842e3f4a7e8889f29f75d3708cea4b260105b68725f not found: ID does not exist"
Oct 30 19:27:39.986600 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.986552    4511 scope.go:122] "RemoveContainer" containerID="6054225f5480b97b1ccef53fb3349285304e0f50d92ed6700ae5a20b5b565f29"
Oct 30 19:27:39.986785 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:27:39.986769    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6054225f5480b97b1ccef53fb3349285304e0f50d92ed6700ae5a20b5b565f29\": container with ID starting with 6054225f5480b97b1ccef53fb3349285304e0f50d92ed6700ae5a20b5b565f29 not found: ID does not exist" containerID="6054225f5480b97b1ccef53fb3349285304e0f50d92ed6700ae5a20b5b565f29"
Oct 30 19:27:39.986827 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.986790    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6054225f5480b97b1ccef53fb3349285304e0f50d92ed6700ae5a20b5b565f29"} err="failed to get container status \"6054225f5480b97b1ccef53fb3349285304e0f50d92ed6700ae5a20b5b565f29\": rpc error: code = NotFound desc = could not find container \"6054225f5480b97b1ccef53fb3349285304e0f50d92ed6700ae5a20b5b565f29\": container with ID starting with 6054225f5480b97b1ccef53fb3349285304e0f50d92ed6700ae5a20b5b565f29 not found: ID does not exist"
Oct 30 19:27:39.986827 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.986805    4511 scope.go:122] "RemoveContainer" containerID="407ec184336f915494a451237550f3de9408f6e0dba83ec3414950910f9f64b4"
Oct 30 19:27:39.987028 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:27:39.987010    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407ec184336f915494a451237550f3de9408f6e0dba83ec3414950910f9f64b4\": container with ID starting with 407ec184336f915494a451237550f3de9408f6e0dba83ec3414950910f9f64b4 not found: ID does not exist" containerID="407ec184336f915494a451237550f3de9408f6e0dba83ec3414950910f9f64b4"
Oct 30 19:27:39.987067 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:39.987038    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407ec184336f915494a451237550f3de9408f6e0dba83ec3414950910f9f64b4"} err="failed to get container status \"407ec184336f915494a451237550f3de9408f6e0dba83ec3414950910f9f64b4\": rpc error: code = NotFound desc = could not find container \"407ec184336f915494a451237550f3de9408f6e0dba83ec3414950910f9f64b4\": container with ID starting with 407ec184336f915494a451237550f3de9408f6e0dba83ec3414950910f9f64b4 not found: ID does not exist"
Oct 30 19:27:40.124314 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:40.124289    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"]
Oct 30 19:27:40.129960 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:40.129935    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-rollback-predictor-7989d9dc9c-dz5ls"]
Oct 30 19:27:40.156024 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:27:40.155999    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:27:41.655551 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:41.655519    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" path="/var/lib/kubelet/pods/9bdd45c6-83a8-4b00-b716-96a72fceda14/volumes"
Oct 30 19:27:44.213615 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.213580    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"]
Oct 30 19:27:44.213984 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.213863    4511 cpu_manager.go:414] "containerMap: removing container" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="storage-initializer"
Oct 30 19:27:44.213984 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.213874    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="storage-initializer"
Oct 30 19:27:44.213984 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.213889    4511 cpu_manager.go:414] "containerMap: removing container" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kube-rbac-proxy"
Oct 30 19:27:44.213984 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.213895    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kube-rbac-proxy"
Oct 30 19:27:44.213984 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.213904    4511 cpu_manager.go:414] "containerMap: removing container" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kserve-container"
Oct 30 19:27:44.213984 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.213909    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kserve-container"
Oct 30 19:27:44.213984 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.213962    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kube-rbac-proxy"
Oct 30 19:27:44.213984 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.213970    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bdd45c6-83a8-4b00-b716-96a72fceda14" containerName="kserve-container"
Oct 30 19:27:44.218718 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.218693    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:44.269648 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.269627    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:27:44.269759 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.269627    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:27:44.269759 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.269690    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:27:44.269759 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.269724    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Oct 30 19:27:44.287018 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.286994    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c352278-0366-4c48-82f9-f1ee43a7e342-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-dffcdc97d-55s7n\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:44.287088 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.287028    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcdbd\" (UniqueName: \"kubernetes.io/projected/2c352278-0366-4c48-82f9-f1ee43a7e342-kube-api-access-dcdbd\") pod \"isvc-canary-stop-v2-predictor-dffcdc97d-55s7n\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:44.287190 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.287172    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c352278-0366-4c48-82f9-f1ee43a7e342-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-dffcdc97d-55s7n\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:44.287234 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.287198    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c352278-0366-4c48-82f9-f1ee43a7e342-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-dffcdc97d-55s7n\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:44.287301 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.287284    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c352278-0366-4c48-82f9-f1ee43a7e342-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-dffcdc97d-55s7n\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:44.317508 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.317487    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Oct 30 19:27:44.317601 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.317509    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:27:44.317662 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.317631    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:27:44.319033 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.319017    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"]
Oct 30 19:27:44.388262 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.388236    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c352278-0366-4c48-82f9-f1ee43a7e342-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-dffcdc97d-55s7n\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:44.388392 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.388266    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c352278-0366-4c48-82f9-f1ee43a7e342-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-dffcdc97d-55s7n\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:44.388392 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.388299    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c352278-0366-4c48-82f9-f1ee43a7e342-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-dffcdc97d-55s7n\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:44.388392 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.388322    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c352278-0366-4c48-82f9-f1ee43a7e342-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-dffcdc97d-55s7n\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:44.388392 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.388337    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcdbd\" (UniqueName: \"kubernetes.io/projected/2c352278-0366-4c48-82f9-f1ee43a7e342-kube-api-access-dcdbd\") pod \"isvc-canary-stop-v2-predictor-dffcdc97d-55s7n\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:44.388744 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.388720    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c352278-0366-4c48-82f9-f1ee43a7e342-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-dffcdc97d-55s7n\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:44.388911 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.388893    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c352278-0366-4c48-82f9-f1ee43a7e342-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-dffcdc97d-55s7n\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:44.389024 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.389004    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c352278-0366-4c48-82f9-f1ee43a7e342-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-dffcdc97d-55s7n\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:44.390853 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.390833    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c352278-0366-4c48-82f9-f1ee43a7e342-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-dffcdc97d-55s7n\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:44.447981 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.447952    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcdbd\" (UniqueName: \"kubernetes.io/projected/2c352278-0366-4c48-82f9-f1ee43a7e342-kube-api-access-dcdbd\") pod \"isvc-canary-stop-v2-predictor-dffcdc97d-55s7n\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:44.536070 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.536002    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:44.960612 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.960570    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"]
Oct 30 19:27:44.967556 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:27:44.967530    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c352278_0366_4c48_82f9_f1ee43a7e342.slice/crio-b92fbb126377dbcbd983cd261b39412d6cb37d54dab1aa4cec2ea9453e941bf3 WatchSource:0}: Error finding container b92fbb126377dbcbd983cd261b39412d6cb37d54dab1aa4cec2ea9453e941bf3: Status 404 returned error can't find the container with id b92fbb126377dbcbd983cd261b39412d6cb37d54dab1aa4cec2ea9453e941bf3
Oct 30 19:27:44.993092 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:44.993061    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" event={"ID":"2c352278-0366-4c48-82f9-f1ee43a7e342","Type":"ContainerStarted","Data":"b92fbb126377dbcbd983cd261b39412d6cb37d54dab1aa4cec2ea9453e941bf3"}
Oct 30 19:27:45.191840 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.191806    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"]
Oct 30 19:27:45.195449 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.195427    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:45.245907 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.245827    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-canary-stop-predictor-serving-cert\""
Oct 30 19:27:45.256539 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.256512    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"]
Oct 30 19:27:45.296169 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.296123    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3947315-9916-4ff5-9301-d853ef395505-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69bd766f7-4f65s\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:45.296169 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.296171    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3947315-9916-4ff5-9301-d853ef395505-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69bd766f7-4f65s\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:45.296366 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.296193    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3947315-9916-4ff5-9301-d853ef395505-cabundle-cert\") pod \"isvc-canary-stop-predictor-69bd766f7-4f65s\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:45.296366 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.296314    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvxzt\" (UniqueName: \"kubernetes.io/projected/a3947315-9916-4ff5-9301-d853ef395505-kube-api-access-bvxzt\") pod \"isvc-canary-stop-predictor-69bd766f7-4f65s\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:45.296490 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.296366    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3947315-9916-4ff5-9301-d853ef395505-proxy-tls\") pod \"isvc-canary-stop-predictor-69bd766f7-4f65s\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:45.396869 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.396840    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvxzt\" (UniqueName: \"kubernetes.io/projected/a3947315-9916-4ff5-9301-d853ef395505-kube-api-access-bvxzt\") pod \"isvc-canary-stop-predictor-69bd766f7-4f65s\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:45.397047 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.396877    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3947315-9916-4ff5-9301-d853ef395505-proxy-tls\") pod \"isvc-canary-stop-predictor-69bd766f7-4f65s\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:45.397047 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.396907    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3947315-9916-4ff5-9301-d853ef395505-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69bd766f7-4f65s\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:45.397047 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.396936    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3947315-9916-4ff5-9301-d853ef395505-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69bd766f7-4f65s\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:45.397047 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.396967    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3947315-9916-4ff5-9301-d853ef395505-cabundle-cert\") pod \"isvc-canary-stop-predictor-69bd766f7-4f65s\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:45.397488 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.397462    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3947315-9916-4ff5-9301-d853ef395505-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69bd766f7-4f65s\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:45.397723 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.397660    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3947315-9916-4ff5-9301-d853ef395505-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69bd766f7-4f65s\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:45.397723 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.397670    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3947315-9916-4ff5-9301-d853ef395505-cabundle-cert\") pod \"isvc-canary-stop-predictor-69bd766f7-4f65s\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:45.400006 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.399985    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3947315-9916-4ff5-9301-d853ef395505-proxy-tls\") pod \"isvc-canary-stop-predictor-69bd766f7-4f65s\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:45.459374 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.459353    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvxzt\" (UniqueName: \"kubernetes.io/projected/a3947315-9916-4ff5-9301-d853ef395505-kube-api-access-bvxzt\") pod \"isvc-canary-stop-predictor-69bd766f7-4f65s\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") " pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:45.514106 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.514046    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:45.944710 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.944678    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"]
Oct 30 19:27:45.952337 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:27:45.952308    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3947315_9916_4ff5_9301_d853ef395505.slice/crio-98d9281fc05235ad2f050cd5636df5a3f33518b48ca0681fc3e713e2f8da54e8 WatchSource:0}: Error finding container 98d9281fc05235ad2f050cd5636df5a3f33518b48ca0681fc3e713e2f8da54e8: Status 404 returned error can't find the container with id 98d9281fc05235ad2f050cd5636df5a3f33518b48ca0681fc3e713e2f8da54e8
Oct 30 19:27:45.999317 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:45.999278    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" event={"ID":"2c352278-0366-4c48-82f9-f1ee43a7e342","Type":"ContainerStarted","Data":"de728e8845ba118dcabcba05c90fb91d341d224c30e5e4ef7a4b547cef1f36cf"}
Oct 30 19:27:46.001046 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:46.001021    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" event={"ID":"a3947315-9916-4ff5-9301-d853ef395505","Type":"ContainerStarted","Data":"98d9281fc05235ad2f050cd5636df5a3f33518b48ca0681fc3e713e2f8da54e8"}
Oct 30 19:27:47.007274 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:47.007236    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" event={"ID":"a3947315-9916-4ff5-9301-d853ef395505","Type":"ContainerStarted","Data":"b663396123bd88839320e61cb15e81938c37c2a97c59ea4033615b102fda0eb5"}
Oct 30 19:27:48.652305 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:48.652272    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:27:49.022857 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:49.022827    4511 generic.go:356] "Generic (PLEG): container finished" podID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerID="de728e8845ba118dcabcba05c90fb91d341d224c30e5e4ef7a4b547cef1f36cf" exitCode=0
Oct 30 19:27:49.023003 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:49.022898    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" event={"ID":"2c352278-0366-4c48-82f9-f1ee43a7e342","Type":"ContainerDied","Data":"de728e8845ba118dcabcba05c90fb91d341d224c30e5e4ef7a4b547cef1f36cf"}
Oct 30 19:27:50.029347 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:50.029260    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" event={"ID":"2c352278-0366-4c48-82f9-f1ee43a7e342","Type":"ContainerStarted","Data":"815efd783811bde14cb4aba27f76863a04ab35dd69a40a971e59c5507b887c8b"}
Oct 30 19:27:50.029347 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:50.029327    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" event={"ID":"2c352278-0366-4c48-82f9-f1ee43a7e342","Type":"ContainerStarted","Data":"055353a86ae0d2e9038a3ccb2e43befe4b5865b394ef4c05296e0e3e518d2d0d"}
Oct 30 19:27:50.029827 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:50.029608    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:50.029827 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:50.029663    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:50.030950 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:50.030929    4511 generic.go:356] "Generic (PLEG): container finished" podID="a3947315-9916-4ff5-9301-d853ef395505" containerID="b663396123bd88839320e61cb15e81938c37c2a97c59ea4033615b102fda0eb5" exitCode=0
Oct 30 19:27:50.031065 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:50.031010    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" event={"ID":"a3947315-9916-4ff5-9301-d853ef395505","Type":"ContainerDied","Data":"b663396123bd88839320e61cb15e81938c37c2a97c59ea4033615b102fda0eb5"}
Oct 30 19:27:50.031327 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:50.031306    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 30 19:27:50.345847 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:50.345795    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" podStartSLOduration=2.4203566260000002 podStartE2EDuration="6.34578197s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.925425344s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:50.227721435 +0000 UTC m=+2964.745406335" watchObservedRunningTime="2026-10-30 19:27:50.34578197 +0000 UTC m=+2964.863466869"
Oct 30 19:27:51.036840 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:51.036803    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" event={"ID":"a3947315-9916-4ff5-9301-d853ef395505","Type":"ContainerStarted","Data":"d91103ce2c097df5c96a7be74311b7a0a30cc1a8e1cc9691113bb836f334c0dd"}
Oct 30 19:27:51.037291 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:51.036868    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" event={"ID":"a3947315-9916-4ff5-9301-d853ef395505","Type":"ContainerStarted","Data":"09b81154f9d0b339e4ea7c2a37f9c12686bef7f1d7d9844fde7efeb44e1f1690"}
Oct 30 19:27:51.037291 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:51.036984    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 30 19:27:51.037291 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:51.037225    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:51.037393 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:51.037369    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:51.038612 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:51.038590    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 30 19:27:51.153614 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:51.153566    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" podStartSLOduration=2.392425906 podStartE2EDuration="6.153553231s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.761127325s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:51.15302103 +0000 UTC m=+2965.670705955" watchObservedRunningTime="2026-10-30 19:27:51.153553231 +0000 UTC m=+2965.671238131"
Oct 30 19:27:52.041167 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:52.041110    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 30 19:27:56.042289 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:56.042262    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:27:56.042793 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:56.042768    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 30 19:27:57.046190 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:57.046157    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:27:57.046672 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:27:57.046648    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 30 19:28:00.748049 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:00.748012    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"]
Oct 30 19:28:00.748579 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:00.748304    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kserve-container" containerID="cri-o://e603518190beb2ca25b21dcebe1eb62e4263d57ff325b4b2c8776e07af6f3979" gracePeriod=30
Oct 30 19:28:00.748579 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:00.748361    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kube-rbac-proxy" containerID="cri-o://9a5e26d58d47b51af69fdabb11efa135532fd6ddad35c8b334b7db6ed3efba80" gracePeriod=30
Oct 30 19:28:01.080338 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:01.080307    4511 generic.go:356] "Generic (PLEG): container finished" podID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerID="9a5e26d58d47b51af69fdabb11efa135532fd6ddad35c8b334b7db6ed3efba80" exitCode=0
Oct 30 19:28:01.080500 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:01.080367    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" event={"ID":"b9039b99-5a63-484f-ac10-1bb91c50a1cb","Type":"ContainerDied","Data":"9a5e26d58d47b51af69fdabb11efa135532fd6ddad35c8b334b7db6ed3efba80"}
Oct 30 19:28:04.711389 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:04.711346    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Oct 30 19:28:05.595588 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:05.595564    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:28:05.755808 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:05.755730    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kube-rbac-proxy-sar-config\") pod \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\" (UID: \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\") "
Oct 30 19:28:05.755808 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:05.755769    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kube-api-access-6w2mk\" (UniqueName: \"kubernetes.io/projected/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kube-api-access-6w2mk\") pod \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\" (UID: \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\") "
Oct 30 19:28:05.756308 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:05.755841    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9039b99-5a63-484f-ac10-1bb91c50a1cb-proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9039b99-5a63-484f-ac10-1bb91c50a1cb-proxy-tls\") pod \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\" (UID: \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\") "
Oct 30 19:28:05.756308 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:05.755864    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kserve-provision-location\") pod \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\" (UID: \"b9039b99-5a63-484f-ac10-1bb91c50a1cb\") "
Oct 30 19:28:05.756308 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:05.756158    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kube-rbac-proxy-sar-config" pod "b9039b99-5a63-484f-ac10-1bb91c50a1cb" (UID: "b9039b99-5a63-484f-ac10-1bb91c50a1cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:05.756308 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:05.756228    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kserve-provision-location" pod "b9039b99-5a63-484f-ac10-1bb91c50a1cb" (UID: "b9039b99-5a63-484f-ac10-1bb91c50a1cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:05.758081 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:05.758052    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9039b99-5a63-484f-ac10-1bb91c50a1cb-proxy-tls" pod "b9039b99-5a63-484f-ac10-1bb91c50a1cb" (UID: "b9039b99-5a63-484f-ac10-1bb91c50a1cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:05.758339 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:05.758089    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kube-api-access-6w2mk" pod "b9039b99-5a63-484f-ac10-1bb91c50a1cb" (UID: "b9039b99-5a63-484f-ac10-1bb91c50a1cb"). InnerVolumeSpecName "kube-api-access-6w2mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:05.856918 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:05.856892    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9039b99-5a63-484f-ac10-1bb91c50a1cb-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:05.856918 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:05.856915    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:05.857157 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:05.856926    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:05.857157 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:05.856935    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w2mk\" (UniqueName: \"kubernetes.io/projected/b9039b99-5a63-484f-ac10-1bb91c50a1cb-kube-api-access-6w2mk\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:06.043539 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:06.043509    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 30 19:28:06.103530 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:06.103499    4511 generic.go:356] "Generic (PLEG): container finished" podID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerID="e603518190beb2ca25b21dcebe1eb62e4263d57ff325b4b2c8776e07af6f3979" exitCode=0
Oct 30 19:28:06.103686 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:06.103551    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" event={"ID":"b9039b99-5a63-484f-ac10-1bb91c50a1cb","Type":"ContainerDied","Data":"e603518190beb2ca25b21dcebe1eb62e4263d57ff325b4b2c8776e07af6f3979"}
Oct 30 19:28:06.103686 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:06.103558    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"
Oct 30 19:28:06.103686 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:06.103611    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p" event={"ID":"b9039b99-5a63-484f-ac10-1bb91c50a1cb","Type":"ContainerDied","Data":"88590e2f420bd43fe8b523cdbdbdc7e0bc1d00082c2a3f45c1649d334cfb4754"}
Oct 30 19:28:06.103686 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:06.103648    4511 scope.go:122] "RemoveContainer" containerID="9a5e26d58d47b51af69fdabb11efa135532fd6ddad35c8b334b7db6ed3efba80"
Oct 30 19:28:06.112047 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:06.112032    4511 scope.go:122] "RemoveContainer" containerID="e603518190beb2ca25b21dcebe1eb62e4263d57ff325b4b2c8776e07af6f3979"
Oct 30 19:28:06.123337 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:06.123315    4511 scope.go:122] "RemoveContainer" containerID="5fca3ce6cf1e722cd4b784d5fbc4ecf869366929cd3f50b0291becda4aeaf659"
Oct 30 19:28:06.130903 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:06.130870    4511 scope.go:122] "RemoveContainer" containerID="9a5e26d58d47b51af69fdabb11efa135532fd6ddad35c8b334b7db6ed3efba80"
Oct 30 19:28:06.131176 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:28:06.131157    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a5e26d58d47b51af69fdabb11efa135532fd6ddad35c8b334b7db6ed3efba80\": container with ID starting with 9a5e26d58d47b51af69fdabb11efa135532fd6ddad35c8b334b7db6ed3efba80 not found: ID does not exist" containerID="9a5e26d58d47b51af69fdabb11efa135532fd6ddad35c8b334b7db6ed3efba80"
Oct 30 19:28:06.131266 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:06.131190    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a5e26d58d47b51af69fdabb11efa135532fd6ddad35c8b334b7db6ed3efba80"} err="failed to get container status \"9a5e26d58d47b51af69fdabb11efa135532fd6ddad35c8b334b7db6ed3efba80\": rpc error: code = NotFound desc = could not find container \"9a5e26d58d47b51af69fdabb11efa135532fd6ddad35c8b334b7db6ed3efba80\": container with ID starting with 9a5e26d58d47b51af69fdabb11efa135532fd6ddad35c8b334b7db6ed3efba80 not found: ID does not exist"
Oct 30 19:28:06.131266 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:06.131213    4511 scope.go:122] "RemoveContainer" containerID="e603518190beb2ca25b21dcebe1eb62e4263d57ff325b4b2c8776e07af6f3979"
Oct 30 19:28:06.131481 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:28:06.131462    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e603518190beb2ca25b21dcebe1eb62e4263d57ff325b4b2c8776e07af6f3979\": container with ID starting with e603518190beb2ca25b21dcebe1eb62e4263d57ff325b4b2c8776e07af6f3979 not found: ID does not exist" containerID="e603518190beb2ca25b21dcebe1eb62e4263d57ff325b4b2c8776e07af6f3979"
Oct 30 19:28:06.131519 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:06.131487    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e603518190beb2ca25b21dcebe1eb62e4263d57ff325b4b2c8776e07af6f3979"} err="failed to get container status \"e603518190beb2ca25b21dcebe1eb62e4263d57ff325b4b2c8776e07af6f3979\": rpc error: code = NotFound desc = could not find container \"e603518190beb2ca25b21dcebe1eb62e4263d57ff325b4b2c8776e07af6f3979\": container with ID starting with e603518190beb2ca25b21dcebe1eb62e4263d57ff325b4b2c8776e07af6f3979 not found: ID does not exist"
Oct 30 19:28:06.131519 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:06.131502    4511 scope.go:122] "RemoveContainer" containerID="5fca3ce6cf1e722cd4b784d5fbc4ecf869366929cd3f50b0291becda4aeaf659"
Oct 30 19:28:06.131727 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:28:06.131709    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fca3ce6cf1e722cd4b784d5fbc4ecf869366929cd3f50b0291becda4aeaf659\": container with ID starting with 5fca3ce6cf1e722cd4b784d5fbc4ecf869366929cd3f50b0291becda4aeaf659 not found: ID does not exist" containerID="5fca3ce6cf1e722cd4b784d5fbc4ecf869366929cd3f50b0291becda4aeaf659"
Oct 30 19:28:06.131787 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:06.131734    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fca3ce6cf1e722cd4b784d5fbc4ecf869366929cd3f50b0291becda4aeaf659"} err="failed to get container status \"5fca3ce6cf1e722cd4b784d5fbc4ecf869366929cd3f50b0291becda4aeaf659\": rpc error: code = NotFound desc = could not find container \"5fca3ce6cf1e722cd4b784d5fbc4ecf869366929cd3f50b0291becda4aeaf659\": container with ID starting with 5fca3ce6cf1e722cd4b784d5fbc4ecf869366929cd3f50b0291becda4aeaf659 not found: ID does not exist"
Oct 30 19:28:06.262289 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:06.262256    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"]
Oct 30 19:28:06.309817 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:06.309755    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-predictor-777575875d-sbr7p"]
Oct 30 19:28:07.047027 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:07.046995    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 30 19:28:07.656433 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:07.656398    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" path="/var/lib/kubelet/pods/b9039b99-5a63-484f-ac10-1bb91c50a1cb/volumes"
Oct 30 19:28:10.156500 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:28:10.156461    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:28:16.043085 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:16.043046    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 30 19:28:17.047283 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:17.047248    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 30 19:28:25.860747 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:25.860643    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 19:28:25.872580 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:25.872538    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 19:28:26.043314 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:26.043275    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 30 19:28:27.046724 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:27.046681    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 30 19:28:36.042737 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:36.042696    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 30 19:28:37.046798 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:37.046758    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 30 19:28:40.156463 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:28:40.156427    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:28:46.043568 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:46.043529    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 30 19:28:47.047108 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:47.047065    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 30 19:28:56.042728 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:56.042689    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 30 19:28:56.651377 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:56.651337    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 30 19:28:57.046754 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:28:57.046721    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 30 19:29:06.651867 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:06.651838    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:29:07.047214 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:07.047185    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:29:10.157050 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:29:10.157007    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:29:13.691470 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:13.691433    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"]
Oct 30 19:29:13.691950 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:13.691692    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kserve-container" containerID="cri-o://055353a86ae0d2e9038a3ccb2e43befe4b5865b394ef4c05296e0e3e518d2d0d" gracePeriod=30
Oct 30 19:29:13.691950 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:13.691726    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kube-rbac-proxy" containerID="cri-o://815efd783811bde14cb4aba27f76863a04ab35dd69a40a971e59c5507b887c8b" gracePeriod=30
Oct 30 19:29:14.198809 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:14.198775    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"]
Oct 30 19:29:14.199048 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:14.199026    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kserve-container" containerID="cri-o://09b81154f9d0b339e4ea7c2a37f9c12686bef7f1d7d9844fde7efeb44e1f1690" gracePeriod=30
Oct 30 19:29:14.199118 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:14.199053    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kube-rbac-proxy" containerID="cri-o://d91103ce2c097df5c96a7be74311b7a0a30cc1a8e1cc9691113bb836f334c0dd" gracePeriod=30
Oct 30 19:29:14.381007 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:14.380974    4511 generic.go:356] "Generic (PLEG): container finished" podID="a3947315-9916-4ff5-9301-d853ef395505" containerID="d91103ce2c097df5c96a7be74311b7a0a30cc1a8e1cc9691113bb836f334c0dd" exitCode=0
Oct 30 19:29:14.381156 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:14.381048    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" event={"ID":"a3947315-9916-4ff5-9301-d853ef395505","Type":"ContainerDied","Data":"d91103ce2c097df5c96a7be74311b7a0a30cc1a8e1cc9691113bb836f334c0dd"}
Oct 30 19:29:14.383196 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:14.383175    4511 generic.go:356] "Generic (PLEG): container finished" podID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerID="815efd783811bde14cb4aba27f76863a04ab35dd69a40a971e59c5507b887c8b" exitCode=0
Oct 30 19:29:14.383288 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:14.383221    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" event={"ID":"2c352278-0366-4c48-82f9-f1ee43a7e342","Type":"ContainerDied","Data":"815efd783811bde14cb4aba27f76863a04ab35dd69a40a971e59c5507b887c8b"}
Oct 30 19:29:16.038105 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:16.038055    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Oct 30 19:29:16.651513 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:16.651476    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Oct 30 19:29:17.042442 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:17.042352    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Oct 30 19:29:17.046753 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:17.046729    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Oct 30 19:29:18.538299 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:18.538273    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:29:18.623008 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:18.622930    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c352278-0366-4c48-82f9-f1ee43a7e342-kube-api-access-dcdbd\" (UniqueName: \"kubernetes.io/projected/2c352278-0366-4c48-82f9-f1ee43a7e342-kube-api-access-dcdbd\") pod \"2c352278-0366-4c48-82f9-f1ee43a7e342\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") "
Oct 30 19:29:18.623008 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:18.622965    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c352278-0366-4c48-82f9-f1ee43a7e342-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c352278-0366-4c48-82f9-f1ee43a7e342-kserve-provision-location\") pod \"2c352278-0366-4c48-82f9-f1ee43a7e342\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") "
Oct 30 19:29:18.623247 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:18.623046    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c352278-0366-4c48-82f9-f1ee43a7e342-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c352278-0366-4c48-82f9-f1ee43a7e342-cabundle-cert\") pod \"2c352278-0366-4c48-82f9-f1ee43a7e342\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") "
Oct 30 19:29:18.623247 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:18.623094    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c352278-0366-4c48-82f9-f1ee43a7e342-proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c352278-0366-4c48-82f9-f1ee43a7e342-proxy-tls\") pod \"2c352278-0366-4c48-82f9-f1ee43a7e342\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") "
Oct 30 19:29:18.623247 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:18.623165    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c352278-0366-4c48-82f9-f1ee43a7e342-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c352278-0366-4c48-82f9-f1ee43a7e342-kube-rbac-proxy-sar-config\") pod \"2c352278-0366-4c48-82f9-f1ee43a7e342\" (UID: \"2c352278-0366-4c48-82f9-f1ee43a7e342\") "
Oct 30 19:29:18.623403 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:18.623322    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c352278-0366-4c48-82f9-f1ee43a7e342-kserve-provision-location" pod "2c352278-0366-4c48-82f9-f1ee43a7e342" (UID: "2c352278-0366-4c48-82f9-f1ee43a7e342"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:18.623452 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:18.623403    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c352278-0366-4c48-82f9-f1ee43a7e342-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:18.623532 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:18.623422    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c352278-0366-4c48-82f9-f1ee43a7e342-cabundle-cert" pod "2c352278-0366-4c48-82f9-f1ee43a7e342" (UID: "2c352278-0366-4c48-82f9-f1ee43a7e342"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:18.623593 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:18.623573    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c352278-0366-4c48-82f9-f1ee43a7e342-kube-rbac-proxy-sar-config" pod "2c352278-0366-4c48-82f9-f1ee43a7e342" (UID: "2c352278-0366-4c48-82f9-f1ee43a7e342"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:18.625352 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:18.625329    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c352278-0366-4c48-82f9-f1ee43a7e342-proxy-tls" pod "2c352278-0366-4c48-82f9-f1ee43a7e342" (UID: "2c352278-0366-4c48-82f9-f1ee43a7e342"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:18.625457 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:18.625422    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c352278-0366-4c48-82f9-f1ee43a7e342-kube-api-access-dcdbd" pod "2c352278-0366-4c48-82f9-f1ee43a7e342" (UID: "2c352278-0366-4c48-82f9-f1ee43a7e342"). InnerVolumeSpecName "kube-api-access-dcdbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:18.723835 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:18.723809    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c352278-0366-4c48-82f9-f1ee43a7e342-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:18.723926 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:18.723843    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c352278-0366-4c48-82f9-f1ee43a7e342-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:18.723926 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:18.723859    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c352278-0366-4c48-82f9-f1ee43a7e342-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:18.723926 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:18.723874    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcdbd\" (UniqueName: \"kubernetes.io/projected/2c352278-0366-4c48-82f9-f1ee43a7e342-kube-api-access-dcdbd\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:18.928782 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:18.928760    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:29:19.026403 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.026378    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3947315-9916-4ff5-9301-d853ef395505-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3947315-9916-4ff5-9301-d853ef395505-cabundle-cert\") pod \"a3947315-9916-4ff5-9301-d853ef395505\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") "
Oct 30 19:29:19.026586 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.026411    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3947315-9916-4ff5-9301-d853ef395505-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3947315-9916-4ff5-9301-d853ef395505-proxy-tls\") pod \"a3947315-9916-4ff5-9301-d853ef395505\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") "
Oct 30 19:29:19.026586 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.026456    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3947315-9916-4ff5-9301-d853ef395505-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3947315-9916-4ff5-9301-d853ef395505-kserve-provision-location\") pod \"a3947315-9916-4ff5-9301-d853ef395505\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") "
Oct 30 19:29:19.026586 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.026499    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3947315-9916-4ff5-9301-d853ef395505-kube-api-access-bvxzt\" (UniqueName: \"kubernetes.io/projected/a3947315-9916-4ff5-9301-d853ef395505-kube-api-access-bvxzt\") pod \"a3947315-9916-4ff5-9301-d853ef395505\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") "
Oct 30 19:29:19.026586 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.026529    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3947315-9916-4ff5-9301-d853ef395505-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3947315-9916-4ff5-9301-d853ef395505-kube-rbac-proxy-sar-config\") pod \"a3947315-9916-4ff5-9301-d853ef395505\" (UID: \"a3947315-9916-4ff5-9301-d853ef395505\") "
Oct 30 19:29:19.026816 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.026782    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3947315-9916-4ff5-9301-d853ef395505-cabundle-cert" pod "a3947315-9916-4ff5-9301-d853ef395505" (UID: "a3947315-9916-4ff5-9301-d853ef395505"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:19.026816 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.026797    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3947315-9916-4ff5-9301-d853ef395505-kserve-provision-location" pod "a3947315-9916-4ff5-9301-d853ef395505" (UID: "a3947315-9916-4ff5-9301-d853ef395505"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:19.027032 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.027004    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3947315-9916-4ff5-9301-d853ef395505-kube-rbac-proxy-sar-config" pod "a3947315-9916-4ff5-9301-d853ef395505" (UID: "a3947315-9916-4ff5-9301-d853ef395505"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:19.028636 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.028617    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3947315-9916-4ff5-9301-d853ef395505-kube-api-access-bvxzt" pod "a3947315-9916-4ff5-9301-d853ef395505" (UID: "a3947315-9916-4ff5-9301-d853ef395505"). InnerVolumeSpecName "kube-api-access-bvxzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:19.028717 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.028648    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3947315-9916-4ff5-9301-d853ef395505-proxy-tls" pod "a3947315-9916-4ff5-9301-d853ef395505" (UID: "a3947315-9916-4ff5-9301-d853ef395505"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:19.127646 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.127619    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3947315-9916-4ff5-9301-d853ef395505-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:19.127646 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.127641    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3947315-9916-4ff5-9301-d853ef395505-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:19.127646 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.127651    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3947315-9916-4ff5-9301-d853ef395505-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:19.127857 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.127660    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3947315-9916-4ff5-9301-d853ef395505-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:19.127857 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.127669    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvxzt\" (UniqueName: \"kubernetes.io/projected/a3947315-9916-4ff5-9301-d853ef395505-kube-api-access-bvxzt\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:19.405600 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.405565    4511 generic.go:356] "Generic (PLEG): container finished" podID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerID="055353a86ae0d2e9038a3ccb2e43befe4b5865b394ef4c05296e0e3e518d2d0d" exitCode=0
Oct 30 19:29:19.405750 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.405613    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"
Oct 30 19:29:19.405750 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.405645    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" event={"ID":"2c352278-0366-4c48-82f9-f1ee43a7e342","Type":"ContainerDied","Data":"055353a86ae0d2e9038a3ccb2e43befe4b5865b394ef4c05296e0e3e518d2d0d"}
Oct 30 19:29:19.405750 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.405703    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n" event={"ID":"2c352278-0366-4c48-82f9-f1ee43a7e342","Type":"ContainerDied","Data":"b92fbb126377dbcbd983cd261b39412d6cb37d54dab1aa4cec2ea9453e941bf3"}
Oct 30 19:29:19.405750 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.405739    4511 scope.go:122] "RemoveContainer" containerID="815efd783811bde14cb4aba27f76863a04ab35dd69a40a971e59c5507b887c8b"
Oct 30 19:29:19.407712 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.407689    4511 generic.go:356] "Generic (PLEG): container finished" podID="a3947315-9916-4ff5-9301-d853ef395505" containerID="09b81154f9d0b339e4ea7c2a37f9c12686bef7f1d7d9844fde7efeb44e1f1690" exitCode=0
Oct 30 19:29:19.407841 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.407742    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" event={"ID":"a3947315-9916-4ff5-9301-d853ef395505","Type":"ContainerDied","Data":"09b81154f9d0b339e4ea7c2a37f9c12686bef7f1d7d9844fde7efeb44e1f1690"}
Oct 30 19:29:19.407841 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.407788    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s" event={"ID":"a3947315-9916-4ff5-9301-d853ef395505","Type":"ContainerDied","Data":"98d9281fc05235ad2f050cd5636df5a3f33518b48ca0681fc3e713e2f8da54e8"}
Oct 30 19:29:19.407841 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.407749    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"
Oct 30 19:29:19.415499 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.415310    4511 scope.go:122] "RemoveContainer" containerID="055353a86ae0d2e9038a3ccb2e43befe4b5865b394ef4c05296e0e3e518d2d0d"
Oct 30 19:29:19.423066 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.423046    4511 scope.go:122] "RemoveContainer" containerID="de728e8845ba118dcabcba05c90fb91d341d224c30e5e4ef7a4b547cef1f36cf"
Oct 30 19:29:19.429907 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.429887    4511 scope.go:122] "RemoveContainer" containerID="815efd783811bde14cb4aba27f76863a04ab35dd69a40a971e59c5507b887c8b"
Oct 30 19:29:19.430176 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:29:19.430152    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815efd783811bde14cb4aba27f76863a04ab35dd69a40a971e59c5507b887c8b\": container with ID starting with 815efd783811bde14cb4aba27f76863a04ab35dd69a40a971e59c5507b887c8b not found: ID does not exist" containerID="815efd783811bde14cb4aba27f76863a04ab35dd69a40a971e59c5507b887c8b"
Oct 30 19:29:19.430246 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.430187    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815efd783811bde14cb4aba27f76863a04ab35dd69a40a971e59c5507b887c8b"} err="failed to get container status \"815efd783811bde14cb4aba27f76863a04ab35dd69a40a971e59c5507b887c8b\": rpc error: code = NotFound desc = could not find container \"815efd783811bde14cb4aba27f76863a04ab35dd69a40a971e59c5507b887c8b\": container with ID starting with 815efd783811bde14cb4aba27f76863a04ab35dd69a40a971e59c5507b887c8b not found: ID does not exist"
Oct 30 19:29:19.430246 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.430202    4511 scope.go:122] "RemoveContainer" containerID="055353a86ae0d2e9038a3ccb2e43befe4b5865b394ef4c05296e0e3e518d2d0d"
Oct 30 19:29:19.430460 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:29:19.430437    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055353a86ae0d2e9038a3ccb2e43befe4b5865b394ef4c05296e0e3e518d2d0d\": container with ID starting with 055353a86ae0d2e9038a3ccb2e43befe4b5865b394ef4c05296e0e3e518d2d0d not found: ID does not exist" containerID="055353a86ae0d2e9038a3ccb2e43befe4b5865b394ef4c05296e0e3e518d2d0d"
Oct 30 19:29:19.430509 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.430467    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055353a86ae0d2e9038a3ccb2e43befe4b5865b394ef4c05296e0e3e518d2d0d"} err="failed to get container status \"055353a86ae0d2e9038a3ccb2e43befe4b5865b394ef4c05296e0e3e518d2d0d\": rpc error: code = NotFound desc = could not find container \"055353a86ae0d2e9038a3ccb2e43befe4b5865b394ef4c05296e0e3e518d2d0d\": container with ID starting with 055353a86ae0d2e9038a3ccb2e43befe4b5865b394ef4c05296e0e3e518d2d0d not found: ID does not exist"
Oct 30 19:29:19.430509 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.430482    4511 scope.go:122] "RemoveContainer" containerID="de728e8845ba118dcabcba05c90fb91d341d224c30e5e4ef7a4b547cef1f36cf"
Oct 30 19:29:19.430687 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:29:19.430672    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de728e8845ba118dcabcba05c90fb91d341d224c30e5e4ef7a4b547cef1f36cf\": container with ID starting with de728e8845ba118dcabcba05c90fb91d341d224c30e5e4ef7a4b547cef1f36cf not found: ID does not exist" containerID="de728e8845ba118dcabcba05c90fb91d341d224c30e5e4ef7a4b547cef1f36cf"
Oct 30 19:29:19.430733 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.430690    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de728e8845ba118dcabcba05c90fb91d341d224c30e5e4ef7a4b547cef1f36cf"} err="failed to get container status \"de728e8845ba118dcabcba05c90fb91d341d224c30e5e4ef7a4b547cef1f36cf\": rpc error: code = NotFound desc = could not find container \"de728e8845ba118dcabcba05c90fb91d341d224c30e5e4ef7a4b547cef1f36cf\": container with ID starting with de728e8845ba118dcabcba05c90fb91d341d224c30e5e4ef7a4b547cef1f36cf not found: ID does not exist"
Oct 30 19:29:19.430733 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.430702    4511 scope.go:122] "RemoveContainer" containerID="d91103ce2c097df5c96a7be74311b7a0a30cc1a8e1cc9691113bb836f334c0dd"
Oct 30 19:29:19.437556 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.437541    4511 scope.go:122] "RemoveContainer" containerID="09b81154f9d0b339e4ea7c2a37f9c12686bef7f1d7d9844fde7efeb44e1f1690"
Oct 30 19:29:19.444308 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.444293    4511 scope.go:122] "RemoveContainer" containerID="b663396123bd88839320e61cb15e81938c37c2a97c59ea4033615b102fda0eb5"
Oct 30 19:29:19.450974 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.450958    4511 scope.go:122] "RemoveContainer" containerID="d91103ce2c097df5c96a7be74311b7a0a30cc1a8e1cc9691113bb836f334c0dd"
Oct 30 19:29:19.451238 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:29:19.451219    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91103ce2c097df5c96a7be74311b7a0a30cc1a8e1cc9691113bb836f334c0dd\": container with ID starting with d91103ce2c097df5c96a7be74311b7a0a30cc1a8e1cc9691113bb836f334c0dd not found: ID does not exist" containerID="d91103ce2c097df5c96a7be74311b7a0a30cc1a8e1cc9691113bb836f334c0dd"
Oct 30 19:29:19.451301 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.451245    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91103ce2c097df5c96a7be74311b7a0a30cc1a8e1cc9691113bb836f334c0dd"} err="failed to get container status \"d91103ce2c097df5c96a7be74311b7a0a30cc1a8e1cc9691113bb836f334c0dd\": rpc error: code = NotFound desc = could not find container \"d91103ce2c097df5c96a7be74311b7a0a30cc1a8e1cc9691113bb836f334c0dd\": container with ID starting with d91103ce2c097df5c96a7be74311b7a0a30cc1a8e1cc9691113bb836f334c0dd not found: ID does not exist"
Oct 30 19:29:19.451301 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.451261    4511 scope.go:122] "RemoveContainer" containerID="09b81154f9d0b339e4ea7c2a37f9c12686bef7f1d7d9844fde7efeb44e1f1690"
Oct 30 19:29:19.451508 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:29:19.451491    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09b81154f9d0b339e4ea7c2a37f9c12686bef7f1d7d9844fde7efeb44e1f1690\": container with ID starting with 09b81154f9d0b339e4ea7c2a37f9c12686bef7f1d7d9844fde7efeb44e1f1690 not found: ID does not exist" containerID="09b81154f9d0b339e4ea7c2a37f9c12686bef7f1d7d9844fde7efeb44e1f1690"
Oct 30 19:29:19.451545 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.451514    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b81154f9d0b339e4ea7c2a37f9c12686bef7f1d7d9844fde7efeb44e1f1690"} err="failed to get container status \"09b81154f9d0b339e4ea7c2a37f9c12686bef7f1d7d9844fde7efeb44e1f1690\": rpc error: code = NotFound desc = could not find container \"09b81154f9d0b339e4ea7c2a37f9c12686bef7f1d7d9844fde7efeb44e1f1690\": container with ID starting with 09b81154f9d0b339e4ea7c2a37f9c12686bef7f1d7d9844fde7efeb44e1f1690 not found: ID does not exist"
Oct 30 19:29:19.451545 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.451528    4511 scope.go:122] "RemoveContainer" containerID="b663396123bd88839320e61cb15e81938c37c2a97c59ea4033615b102fda0eb5"
Oct 30 19:29:19.451754 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:29:19.451740    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b663396123bd88839320e61cb15e81938c37c2a97c59ea4033615b102fda0eb5\": container with ID starting with b663396123bd88839320e61cb15e81938c37c2a97c59ea4033615b102fda0eb5 not found: ID does not exist" containerID="b663396123bd88839320e61cb15e81938c37c2a97c59ea4033615b102fda0eb5"
Oct 30 19:29:19.451807 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.451757    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b663396123bd88839320e61cb15e81938c37c2a97c59ea4033615b102fda0eb5"} err="failed to get container status \"b663396123bd88839320e61cb15e81938c37c2a97c59ea4033615b102fda0eb5\": rpc error: code = NotFound desc = could not find container \"b663396123bd88839320e61cb15e81938c37c2a97c59ea4033615b102fda0eb5\": container with ID starting with b663396123bd88839320e61cb15e81938c37c2a97c59ea4033615b102fda0eb5 not found: ID does not exist"
Oct 30 19:29:19.564183 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.564160    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"]
Oct 30 19:29:19.568435 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.568415    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-v2-predictor-dffcdc97d-55s7n"]
Oct 30 19:29:19.656363 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.656303    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" path="/var/lib/kubelet/pods/2c352278-0366-4c48-82f9-f1ee43a7e342/volumes"
Oct 30 19:29:19.673621 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.673597    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"]
Oct 30 19:29:19.677702 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:19.677683    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-canary-stop-predictor-69bd766f7-4f65s"]
Oct 30 19:29:21.655884 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:21.655846    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3947315-9916-4ff5-9301-d853ef395505" path="/var/lib/kubelet/pods/a3947315-9916-4ff5-9301-d853ef395505/volumes"
Oct 30 19:29:22.444459 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444427    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"]
Oct 30 19:29:22.444706 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444694    4511 cpu_manager.go:414] "containerMap: removing container" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kserve-container"
Oct 30 19:29:22.444752 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444707    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kserve-container"
Oct 30 19:29:22.444752 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444720    4511 cpu_manager.go:414] "containerMap: removing container" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="storage-initializer"
Oct 30 19:29:22.444752 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444726    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="storage-initializer"
Oct 30 19:29:22.444752 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444732    4511 cpu_manager.go:414] "containerMap: removing container" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="storage-initializer"
Oct 30 19:29:22.444752 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444737    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="storage-initializer"
Oct 30 19:29:22.444752 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444742    4511 cpu_manager.go:414] "containerMap: removing container" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kserve-container"
Oct 30 19:29:22.444752 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444746    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kserve-container"
Oct 30 19:29:22.444957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444757    4511 cpu_manager.go:414] "containerMap: removing container" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kube-rbac-proxy"
Oct 30 19:29:22.444957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444761    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kube-rbac-proxy"
Oct 30 19:29:22.444957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444767    4511 cpu_manager.go:414] "containerMap: removing container" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="storage-initializer"
Oct 30 19:29:22.444957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444771    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="storage-initializer"
Oct 30 19:29:22.444957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444777    4511 cpu_manager.go:414] "containerMap: removing container" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kube-rbac-proxy"
Oct 30 19:29:22.444957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444781    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kube-rbac-proxy"
Oct 30 19:29:22.444957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444792    4511 cpu_manager.go:414] "containerMap: removing container" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kserve-container"
Oct 30 19:29:22.444957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444797    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kserve-container"
Oct 30 19:29:22.444957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444805    4511 cpu_manager.go:414] "containerMap: removing container" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kube-rbac-proxy"
Oct 30 19:29:22.444957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444810    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kube-rbac-proxy"
Oct 30 19:29:22.444957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444855    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kube-rbac-proxy"
Oct 30 19:29:22.444957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444862    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kube-rbac-proxy"
Oct 30 19:29:22.444957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444869    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3947315-9916-4ff5-9301-d853ef395505" containerName="kserve-container"
Oct 30 19:29:22.444957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444875    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kube-rbac-proxy"
Oct 30 19:29:22.444957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444881    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9039b99-5a63-484f-ac10-1bb91c50a1cb" containerName="kserve-container"
Oct 30 19:29:22.444957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.444889    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c352278-0366-4c48-82f9-f1ee43a7e342" containerName="kserve-container"
Oct 30 19:29:22.449913 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.449898    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:22.500414 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.500385    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:29:22.547606 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.547580    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:29:22.547606 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.547596    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:29:22.547773 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.547613    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:29:22.547773 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.547620    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-lightgbm-predictor-serving-cert\""
Oct 30 19:29:22.547773 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.547613    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Oct 30 19:29:22.547773 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.547659    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:29:22.549977 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.549959    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"]
Oct 30 19:29:22.555594 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.555566    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvvsc\" (UniqueName: \"kubernetes.io/projected/42130919-0610-4230-9e13-cf4fb6ec1ec3-kube-api-access-hvvsc\") pod \"isvc-lightgbm-predictor-575fb6bc48-gk9w7\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:22.555721 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.555621    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42130919-0610-4230-9e13-cf4fb6ec1ec3-proxy-tls\") pod \"isvc-lightgbm-predictor-575fb6bc48-gk9w7\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:22.555794 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.555729    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42130919-0610-4230-9e13-cf4fb6ec1ec3-cabundle-cert\") pod \"isvc-lightgbm-predictor-575fb6bc48-gk9w7\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:22.555794 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.555780    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42130919-0610-4230-9e13-cf4fb6ec1ec3-kserve-provision-location\") pod \"isvc-lightgbm-predictor-575fb6bc48-gk9w7\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:22.555874 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.555858    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42130919-0610-4230-9e13-cf4fb6ec1ec3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-575fb6bc48-gk9w7\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:22.656762 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.656717    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42130919-0610-4230-9e13-cf4fb6ec1ec3-proxy-tls\") pod \"isvc-lightgbm-predictor-575fb6bc48-gk9w7\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:22.657275 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.656775    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42130919-0610-4230-9e13-cf4fb6ec1ec3-cabundle-cert\") pod \"isvc-lightgbm-predictor-575fb6bc48-gk9w7\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:22.657275 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.656811    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42130919-0610-4230-9e13-cf4fb6ec1ec3-kserve-provision-location\") pod \"isvc-lightgbm-predictor-575fb6bc48-gk9w7\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:22.657275 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.656830    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42130919-0610-4230-9e13-cf4fb6ec1ec3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-575fb6bc48-gk9w7\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:22.657275 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.656861    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvvsc\" (UniqueName: \"kubernetes.io/projected/42130919-0610-4230-9e13-cf4fb6ec1ec3-kube-api-access-hvvsc\") pod \"isvc-lightgbm-predictor-575fb6bc48-gk9w7\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:22.657460 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.657308    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42130919-0610-4230-9e13-cf4fb6ec1ec3-kserve-provision-location\") pod \"isvc-lightgbm-predictor-575fb6bc48-gk9w7\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:22.657511 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.657488    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42130919-0610-4230-9e13-cf4fb6ec1ec3-cabundle-cert\") pod \"isvc-lightgbm-predictor-575fb6bc48-gk9w7\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:22.657548 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.657505    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42130919-0610-4230-9e13-cf4fb6ec1ec3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-575fb6bc48-gk9w7\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:22.659393 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.659368    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42130919-0610-4230-9e13-cf4fb6ec1ec3-proxy-tls\") pod \"isvc-lightgbm-predictor-575fb6bc48-gk9w7\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:22.716537 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.716472    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvvsc\" (UniqueName: \"kubernetes.io/projected/42130919-0610-4230-9e13-cf4fb6ec1ec3-kube-api-access-hvvsc\") pod \"isvc-lightgbm-predictor-575fb6bc48-gk9w7\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:22.766372 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:22.766346    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:23.188550 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:23.188517    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"]
Oct 30 19:29:23.195671 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:29:23.195640    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42130919_0610_4230_9e13_cf4fb6ec1ec3.slice/crio-58ea6bcc376b655ea34f8028ae02dec4dd1fcf0cffa1aeb818940b16d4c85788 WatchSource:0}: Error finding container 58ea6bcc376b655ea34f8028ae02dec4dd1fcf0cffa1aeb818940b16d4c85788: Status 404 returned error can't find the container with id 58ea6bcc376b655ea34f8028ae02dec4dd1fcf0cffa1aeb818940b16d4c85788
Oct 30 19:29:23.426335 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:23.426295    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" event={"ID":"42130919-0610-4230-9e13-cf4fb6ec1ec3","Type":"ContainerStarted","Data":"53243cf7a939021235b5110397c618822d07967206b18c08532d3b00878ccc0e"}
Oct 30 19:29:23.426499 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:23.426377    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" event={"ID":"42130919-0610-4230-9e13-cf4fb6ec1ec3","Type":"ContainerStarted","Data":"58ea6bcc376b655ea34f8028ae02dec4dd1fcf0cffa1aeb818940b16d4c85788"}
Oct 30 19:29:27.441335 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:27.441254    4511 generic.go:356] "Generic (PLEG): container finished" podID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerID="53243cf7a939021235b5110397c618822d07967206b18c08532d3b00878ccc0e" exitCode=0
Oct 30 19:29:27.441686 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:27.441329    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" event={"ID":"42130919-0610-4230-9e13-cf4fb6ec1ec3","Type":"ContainerDied","Data":"53243cf7a939021235b5110397c618822d07967206b18c08532d3b00878ccc0e"}
Oct 30 19:29:35.475228 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:35.475181    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" event={"ID":"42130919-0610-4230-9e13-cf4fb6ec1ec3","Type":"ContainerStarted","Data":"48b4584d8621c5765b70db22d9927848987d8096f319179ffe286ba534f3e1a9"}
Oct 30 19:29:35.475649 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:35.475301    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" event={"ID":"42130919-0610-4230-9e13-cf4fb6ec1ec3","Type":"ContainerStarted","Data":"3290f9f463a8ac1cb166b92011f484c0ec3e3547c5009383005eb33862a065e5"}
Oct 30 19:29:35.475649 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:35.475577    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:35.475775 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:35.475757    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:35.476786 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:35.476761    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 30 19:29:35.617010 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:35.616963    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" podStartSLOduration=2.207246903 podStartE2EDuration="13.616951866s" totalImagesPullingTime="7.575852256s" totalInitContainerRuntime="3.833852707s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:35.615951945 +0000 UTC m=+3070.133636878" watchObservedRunningTime="2026-10-30 19:29:35.616951866 +0000 UTC m=+3070.134636766"
Oct 30 19:29:36.479099 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:36.479064    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 30 19:29:37.482866 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:37.482829    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 30 19:29:40.156246 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:29:40.156210    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:29:42.487643 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.487614    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:29:42.488204 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.488176    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 30 19:29:42.502023 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.501993    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"]
Oct 30 19:29:42.505547 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.505523    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"
Oct 30 19:29:42.559841 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.559814    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"storage-config\""
Oct 30 19:29:42.559999 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.559820    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Oct 30 19:29:42.559999 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.559894    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:29:42.590526 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.590489    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:29:42.595735 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.595714    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Oct 30 19:29:42.595805 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.595781    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:29:42.606137 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.606110    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90295502-08aa-43e6-a211-74d45f4147cc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f\" (UID: \"90295502-08aa-43e6-a211-74d45f4147cc\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"
Oct 30 19:29:42.606277 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.606163    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90295502-08aa-43e6-a211-74d45f4147cc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f\" (UID: \"90295502-08aa-43e6-a211-74d45f4147cc\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"
Oct 30 19:29:42.606277 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.606189    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90295502-08aa-43e6-a211-74d45f4147cc-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f\" (UID: \"90295502-08aa-43e6-a211-74d45f4147cc\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"
Oct 30 19:29:42.606394 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.606373    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7fms\" (UniqueName: \"kubernetes.io/projected/90295502-08aa-43e6-a211-74d45f4147cc-kube-api-access-l7fms\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f\" (UID: \"90295502-08aa-43e6-a211-74d45f4147cc\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"
Oct 30 19:29:42.607266 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.607246    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"]
Oct 30 19:29:42.706957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.706918    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90295502-08aa-43e6-a211-74d45f4147cc-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f\" (UID: \"90295502-08aa-43e6-a211-74d45f4147cc\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"
Oct 30 19:29:42.707187 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.707000    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7fms\" (UniqueName: \"kubernetes.io/projected/90295502-08aa-43e6-a211-74d45f4147cc-kube-api-access-l7fms\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f\" (UID: \"90295502-08aa-43e6-a211-74d45f4147cc\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"
Oct 30 19:29:42.707187 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.707051    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90295502-08aa-43e6-a211-74d45f4147cc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f\" (UID: \"90295502-08aa-43e6-a211-74d45f4147cc\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"
Oct 30 19:29:42.707187 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.707081    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90295502-08aa-43e6-a211-74d45f4147cc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f\" (UID: \"90295502-08aa-43e6-a211-74d45f4147cc\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"
Oct 30 19:29:42.707617 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.707592    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90295502-08aa-43e6-a211-74d45f4147cc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f\" (UID: \"90295502-08aa-43e6-a211-74d45f4147cc\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"
Oct 30 19:29:42.707889 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.707862    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90295502-08aa-43e6-a211-74d45f4147cc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f\" (UID: \"90295502-08aa-43e6-a211-74d45f4147cc\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"
Oct 30 19:29:42.709637 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.709620    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90295502-08aa-43e6-a211-74d45f4147cc-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f\" (UID: \"90295502-08aa-43e6-a211-74d45f4147cc\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"
Oct 30 19:29:42.765796 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.765732    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7fms\" (UniqueName: \"kubernetes.io/projected/90295502-08aa-43e6-a211-74d45f4147cc-kube-api-access-l7fms\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f\" (UID: \"90295502-08aa-43e6-a211-74d45f4147cc\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"
Oct 30 19:29:42.830035 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:42.829999    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"
Oct 30 19:29:43.250786 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:43.250751    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"]
Oct 30 19:29:43.257494 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:29:43.257466    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90295502_08aa_43e6_a211_74d45f4147cc.slice/crio-7ff61de6cf14cc24bb016678b509b293cf679c03c5c649313c4c1677dddeaf85 WatchSource:0}: Error finding container 7ff61de6cf14cc24bb016678b509b293cf679c03c5c649313c4c1677dddeaf85: Status 404 returned error can't find the container with id 7ff61de6cf14cc24bb016678b509b293cf679c03c5c649313c4c1677dddeaf85
Oct 30 19:29:43.504785 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:43.504703    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f" event={"ID":"90295502-08aa-43e6-a211-74d45f4147cc","Type":"ContainerStarted","Data":"3f8984d0b9658a2ab75dd44c363d5db96c093e0c8a89f716d204119786fb7b43"}
Oct 30 19:29:43.504785 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:43.504776    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f" event={"ID":"90295502-08aa-43e6-a211-74d45f4147cc","Type":"ContainerStarted","Data":"7ff61de6cf14cc24bb016678b509b293cf679c03c5c649313c4c1677dddeaf85"}
Oct 30 19:29:47.522704 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:47.522629    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f_90295502-08aa-43e6-a211-74d45f4147cc/storage-initializer/0.log"
Oct 30 19:29:47.522704 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:47.522665    4511 generic.go:356] "Generic (PLEG): container finished" podID="90295502-08aa-43e6-a211-74d45f4147cc" containerID="3f8984d0b9658a2ab75dd44c363d5db96c093e0c8a89f716d204119786fb7b43" exitCode=1
Oct 30 19:29:47.523122 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:47.522743    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f" event={"ID":"90295502-08aa-43e6-a211-74d45f4147cc","Type":"ContainerDied","Data":"3f8984d0b9658a2ab75dd44c363d5db96c093e0c8a89f716d204119786fb7b43"}
Oct 30 19:29:48.529227 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:48.529196    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f_90295502-08aa-43e6-a211-74d45f4147cc/storage-initializer/0.log"
Oct 30 19:29:48.529802 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:48.529254    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f" event={"ID":"90295502-08aa-43e6-a211-74d45f4147cc","Type":"ContainerStarted","Data":"e28313c203c77c334da174f2a73b347090ec7b454dad3f688f636e3b0e9d1da7"}
Oct 30 19:29:51.541732 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:51.541706    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f_90295502-08aa-43e6-a211-74d45f4147cc/storage-initializer/1.log"
Oct 30 19:29:51.542122 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:51.542093    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f_90295502-08aa-43e6-a211-74d45f4147cc/storage-initializer/0.log"
Oct 30 19:29:51.542201 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:51.542143    4511 generic.go:356] "Generic (PLEG): container finished" podID="90295502-08aa-43e6-a211-74d45f4147cc" containerID="e28313c203c77c334da174f2a73b347090ec7b454dad3f688f636e3b0e9d1da7" exitCode=1
Oct 30 19:29:51.542237 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:51.542198    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f" event={"ID":"90295502-08aa-43e6-a211-74d45f4147cc","Type":"ContainerDied","Data":"e28313c203c77c334da174f2a73b347090ec7b454dad3f688f636e3b0e9d1da7"}
Oct 30 19:29:51.542271 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:51.542259    4511 scope.go:122] "RemoveContainer" containerID="3f8984d0b9658a2ab75dd44c363d5db96c093e0c8a89f716d204119786fb7b43"
Oct 30 19:29:51.542583 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:51.542555    4511 scope.go:122] "RemoveContainer" containerID="3f8984d0b9658a2ab75dd44c363d5db96c093e0c8a89f716d204119786fb7b43"
Oct 30 19:29:51.553283 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:29:51.553252    4511 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f_e2e-gw1-4a3552942a(90295502-08aa-43e6-a211-74d45f4147cc)\"" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f" podUID="90295502-08aa-43e6-a211-74d45f4147cc"
Oct 30 19:29:51.553451 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:29:51.553420    4511 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f_e2e-gw1-4a3552942a_90295502-08aa-43e6-a211-74d45f4147cc_0 in pod sandbox 7ff61de6cf14cc24bb016678b509b293cf679c03c5c649313c4c1677dddeaf85 from index: no such id: '3f8984d0b9658a2ab75dd44c363d5db96c093e0c8a89f716d204119786fb7b43'" containerID="3f8984d0b9658a2ab75dd44c363d5db96c093e0c8a89f716d204119786fb7b43"
Oct 30 19:29:51.553527 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:51.553462    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8984d0b9658a2ab75dd44c363d5db96c093e0c8a89f716d204119786fb7b43"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f_e2e-gw1-4a3552942a_90295502-08aa-43e6-a211-74d45f4147cc_0 in pod sandbox 7ff61de6cf14cc24bb016678b509b293cf679c03c5c649313c4c1677dddeaf85 from index: no such id: '3f8984d0b9658a2ab75dd44c363d5db96c093e0c8a89f716d204119786fb7b43'"
Oct 30 19:29:52.335542 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:52.335515    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"]
Oct 30 19:29:52.488360 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:52.488328    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 30 19:29:52.547383 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:52.547360    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f_90295502-08aa-43e6-a211-74d45f4147cc/storage-initializer/1.log"
Oct 30 19:29:52.665774 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:52.665754    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f_90295502-08aa-43e6-a211-74d45f4147cc/storage-initializer/1.log"
Oct 30 19:29:52.669213 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:52.669194    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"
Oct 30 19:29:52.785095 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:52.785071    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90295502-08aa-43e6-a211-74d45f4147cc-proxy-tls\" (UniqueName: \"kubernetes.io/secret/90295502-08aa-43e6-a211-74d45f4147cc-proxy-tls\") pod \"90295502-08aa-43e6-a211-74d45f4147cc\" (UID: \"90295502-08aa-43e6-a211-74d45f4147cc\") "
Oct 30 19:29:52.785236 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:52.785107    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90295502-08aa-43e6-a211-74d45f4147cc-kube-api-access-l7fms\" (UniqueName: \"kubernetes.io/projected/90295502-08aa-43e6-a211-74d45f4147cc-kube-api-access-l7fms\") pod \"90295502-08aa-43e6-a211-74d45f4147cc\" (UID: \"90295502-08aa-43e6-a211-74d45f4147cc\") "
Oct 30 19:29:52.785236 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:52.785147    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90295502-08aa-43e6-a211-74d45f4147cc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90295502-08aa-43e6-a211-74d45f4147cc-kube-rbac-proxy-sar-config\") pod \"90295502-08aa-43e6-a211-74d45f4147cc\" (UID: \"90295502-08aa-43e6-a211-74d45f4147cc\") "
Oct 30 19:29:52.785236 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:52.785192    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90295502-08aa-43e6-a211-74d45f4147cc-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90295502-08aa-43e6-a211-74d45f4147cc-kserve-provision-location\") pod \"90295502-08aa-43e6-a211-74d45f4147cc\" (UID: \"90295502-08aa-43e6-a211-74d45f4147cc\") "
Oct 30 19:29:52.785557 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:52.785532    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90295502-08aa-43e6-a211-74d45f4147cc-kube-rbac-proxy-sar-config" pod "90295502-08aa-43e6-a211-74d45f4147cc" (UID: "90295502-08aa-43e6-a211-74d45f4147cc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:52.785557 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:52.785531    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90295502-08aa-43e6-a211-74d45f4147cc-kserve-provision-location" pod "90295502-08aa-43e6-a211-74d45f4147cc" (UID: "90295502-08aa-43e6-a211-74d45f4147cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:52.787436 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:52.787408    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90295502-08aa-43e6-a211-74d45f4147cc-kube-api-access-l7fms" pod "90295502-08aa-43e6-a211-74d45f4147cc" (UID: "90295502-08aa-43e6-a211-74d45f4147cc"). InnerVolumeSpecName "kube-api-access-l7fms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:52.787545 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:52.787509    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90295502-08aa-43e6-a211-74d45f4147cc-proxy-tls" pod "90295502-08aa-43e6-a211-74d45f4147cc" (UID: "90295502-08aa-43e6-a211-74d45f4147cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:52.886057 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:52.886002    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90295502-08aa-43e6-a211-74d45f4147cc-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:52.886057 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:52.886024    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90295502-08aa-43e6-a211-74d45f4147cc-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:52.886057 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:52.886035    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7fms\" (UniqueName: \"kubernetes.io/projected/90295502-08aa-43e6-a211-74d45f4147cc-kube-api-access-l7fms\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:52.886057 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:52.886043    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90295502-08aa-43e6-a211-74d45f4147cc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:53.552149 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:53.552104    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-4a3552942a_isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f_90295502-08aa-43e6-a211-74d45f4147cc/storage-initializer/1.log"
Oct 30 19:29:53.552558 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:53.552215    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f" event={"ID":"90295502-08aa-43e6-a211-74d45f4147cc","Type":"ContainerDied","Data":"7ff61de6cf14cc24bb016678b509b293cf679c03c5c649313c4c1677dddeaf85"}
Oct 30 19:29:53.552558 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:53.552223    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"
Oct 30 19:29:53.552558 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:53.552269    4511 scope.go:122] "RemoveContainer" containerID="e28313c203c77c334da174f2a73b347090ec7b454dad3f688f636e3b0e9d1da7"
Oct 30 19:29:53.821555 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:53.821478    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"]
Oct 30 19:29:53.825937 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:53.825911    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-global-fail-predictor-5f8f47fb6b-drb7f"]
Oct 30 19:29:55.655791 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:55.655750    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90295502-08aa-43e6-a211-74d45f4147cc" path="/var/lib/kubelet/pods/90295502-08aa-43e6-a211-74d45f4147cc/volumes"
Oct 30 19:29:57.266410 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.266379    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"]
Oct 30 19:29:57.266789 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.266681    4511 cpu_manager.go:414] "containerMap: removing container" podUID="90295502-08aa-43e6-a211-74d45f4147cc" containerName="storage-initializer"
Oct 30 19:29:57.266789 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.266692    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90295502-08aa-43e6-a211-74d45f4147cc" containerName="storage-initializer"
Oct 30 19:29:57.266789 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.266715    4511 cpu_manager.go:414] "containerMap: removing container" podUID="90295502-08aa-43e6-a211-74d45f4147cc" containerName="storage-initializer"
Oct 30 19:29:57.266789 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.266720    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90295502-08aa-43e6-a211-74d45f4147cc" containerName="storage-initializer"
Oct 30 19:29:57.266789 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.266769    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="90295502-08aa-43e6-a211-74d45f4147cc" containerName="storage-initializer"
Oct 30 19:29:57.266789 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.266792    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="90295502-08aa-43e6-a211-74d45f4147cc" containerName="storage-initializer"
Oct 30 19:29:57.277492 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.277462    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:29:57.316038 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.316005    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba7ed3a8-60b0-43fb-8485-60c50d772754-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:29:57.316156 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.316051    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba7ed3a8-60b0-43fb-8485-60c50d772754-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:29:57.316264 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.316238    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba7ed3a8-60b0-43fb-8485-60c50d772754-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:29:57.316327 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.316286    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fv7q\" (UniqueName: \"kubernetes.io/projected/ba7ed3a8-60b0-43fb-8485-60c50d772754-kube-api-access-8fv7q\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:29:57.316421 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.316400    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba7ed3a8-60b0-43fb-8485-60c50d772754-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:29:57.327835 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.327805    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"storage-config\""
Oct 30 19:29:57.375071 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.375052    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:29:57.375202 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.375099    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Oct 30 19:29:57.375202 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.375170    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:29:57.375202 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.375169    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Oct 30 19:29:57.375202 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.375186    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:29:57.375349 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.375186    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:29:57.377296 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.377269    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"]
Oct 30 19:29:57.417679 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.417651    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba7ed3a8-60b0-43fb-8485-60c50d772754-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:29:57.417769 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.417719    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba7ed3a8-60b0-43fb-8485-60c50d772754-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:29:57.417769 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.417754    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba7ed3a8-60b0-43fb-8485-60c50d772754-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:29:57.417853 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.417836    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba7ed3a8-60b0-43fb-8485-60c50d772754-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:29:57.417898 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.417869    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fv7q\" (UniqueName: \"kubernetes.io/projected/ba7ed3a8-60b0-43fb-8485-60c50d772754-kube-api-access-8fv7q\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:29:57.418227 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.418193    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba7ed3a8-60b0-43fb-8485-60c50d772754-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:29:57.418566 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.418543    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba7ed3a8-60b0-43fb-8485-60c50d772754-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:29:57.418763 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.418743    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba7ed3a8-60b0-43fb-8485-60c50d772754-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:29:57.420354 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.420335    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba7ed3a8-60b0-43fb-8485-60c50d772754-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:29:57.477297 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.477266    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fv7q\" (UniqueName: \"kubernetes.io/projected/ba7ed3a8-60b0-43fb-8485-60c50d772754-kube-api-access-8fv7q\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:29:57.630237 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:57.630212    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:29:58.060915 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:58.060881    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"]
Oct 30 19:29:58.068430 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:29:58.068387    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7ed3a8_60b0_43fb_8485_60c50d772754.slice/crio-171baa23114ede257863b4d59bbc4f084e4ff5d696342c612e246e0c339eb847 WatchSource:0}: Error finding container 171baa23114ede257863b4d59bbc4f084e4ff5d696342c612e246e0c339eb847: Status 404 returned error can't find the container with id 171baa23114ede257863b4d59bbc4f084e4ff5d696342c612e246e0c339eb847
Oct 30 19:29:58.573036 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:58.573001    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" event={"ID":"ba7ed3a8-60b0-43fb-8485-60c50d772754","Type":"ContainerStarted","Data":"4798ca25c5c05fe40e21e542267aaa9cd6e5ed1c6e5fbaef905e17588b047f0c"}
Oct 30 19:29:58.573441 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:58.573066    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" event={"ID":"ba7ed3a8-60b0-43fb-8485-60c50d772754","Type":"ContainerStarted","Data":"171baa23114ede257863b4d59bbc4f084e4ff5d696342c612e246e0c339eb847"}
Oct 30 19:29:59.579363 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:59.579326    4511 generic.go:356] "Generic (PLEG): container finished" podID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerID="4798ca25c5c05fe40e21e542267aaa9cd6e5ed1c6e5fbaef905e17588b047f0c" exitCode=0
Oct 30 19:29:59.579781 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:29:59.579403    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" event={"ID":"ba7ed3a8-60b0-43fb-8485-60c50d772754","Type":"ContainerDied","Data":"4798ca25c5c05fe40e21e542267aaa9cd6e5ed1c6e5fbaef905e17588b047f0c"}
Oct 30 19:30:00.585718 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:00.585682    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" event={"ID":"ba7ed3a8-60b0-43fb-8485-60c50d772754","Type":"ContainerStarted","Data":"d6fcd4aa7f81359515b844d90b71fcba5d57fdd84ff4d9461089b8478f0f0ca0"}
Oct 30 19:30:00.586115 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:00.585765    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" event={"ID":"ba7ed3a8-60b0-43fb-8485-60c50d772754","Type":"ContainerStarted","Data":"ff3e0846b7eb9963ecd0a847bd9a18545306b49aee49ddb9e7d3707d1a4e8f6a"}
Oct 30 19:30:00.586115 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:00.585835    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:30:01.590095 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:01.590067    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:30:01.591442 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:01.591415    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Oct 30 19:30:02.488924 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:02.488888    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 30 19:30:02.594117 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:02.594080    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Oct 30 19:30:07.598938 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:07.598911    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:30:07.599418 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:07.599392    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Oct 30 19:30:07.762660 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:07.762602    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" podStartSLOduration=9.748007089 podStartE2EDuration="10.762584583s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.014577494s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:00.702088497 +0000 UTC m=+3095.219773436" watchObservedRunningTime="2026-10-30 19:30:07.762584583 +0000 UTC m=+3102.280269487"
Oct 30 19:30:10.156087 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:30:10.156052    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:30:12.488363 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:12.488324    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 30 19:30:17.600182 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:17.600142    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Oct 30 19:30:22.488537 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:22.488500    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 30 19:30:27.599727 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:27.599683    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Oct 30 19:30:32.488354 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:32.488317    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 30 19:30:37.600324 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:37.600285    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Oct 30 19:30:40.155671 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:30:40.155635    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:30:42.488634 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:42.488589    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 30 19:30:47.599384 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:47.599342    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Oct 30 19:30:52.488596 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:52.488553    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 30 19:30:57.599368 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:30:57.599327    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Oct 30 19:31:02.489073 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:02.489020    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Oct 30 19:31:07.600111 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:07.600069    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Oct 30 19:31:10.156370 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:31:10.156328    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:31:11.656475 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:11.656445    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:31:14.651985 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:14.651946    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:31:16.995181 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:16.995147    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"]
Oct 30 19:31:16.995588 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:16.995375    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kserve-container" containerID="cri-o://ff3e0846b7eb9963ecd0a847bd9a18545306b49aee49ddb9e7d3707d1a4e8f6a" gracePeriod=30
Oct 30 19:31:16.995588 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:16.995429    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kube-rbac-proxy" containerID="cri-o://d6fcd4aa7f81359515b844d90b71fcba5d57fdd84ff4d9461089b8478f0f0ca0" gracePeriod=30
Oct 30 19:31:17.594419 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:17.594382    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.59:8643/healthz\": dial tcp 10.133.0.59:8643: connect: connection refused"
Oct 30 19:31:17.884238 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:17.884164    4511 generic.go:356] "Generic (PLEG): container finished" podID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerID="d6fcd4aa7f81359515b844d90b71fcba5d57fdd84ff4d9461089b8478f0f0ca0" exitCode=0
Oct 30 19:31:17.884238 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:17.884215    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" event={"ID":"ba7ed3a8-60b0-43fb-8485-60c50d772754","Type":"ContainerDied","Data":"d6fcd4aa7f81359515b844d90b71fcba5d57fdd84ff4d9461089b8478f0f0ca0"}
Oct 30 19:31:21.841743 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.841720    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:31:21.879069 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.879051    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7ed3a8-60b0-43fb-8485-60c50d772754-kube-api-access-8fv7q\" (UniqueName: \"kubernetes.io/projected/ba7ed3a8-60b0-43fb-8485-60c50d772754-kube-api-access-8fv7q\") pod \"ba7ed3a8-60b0-43fb-8485-60c50d772754\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") "
Oct 30 19:31:21.879174 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.879090    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7ed3a8-60b0-43fb-8485-60c50d772754-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba7ed3a8-60b0-43fb-8485-60c50d772754-kserve-provision-location\") pod \"ba7ed3a8-60b0-43fb-8485-60c50d772754\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") "
Oct 30 19:31:21.879220 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.879199    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba7ed3a8-60b0-43fb-8485-60c50d772754-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba7ed3a8-60b0-43fb-8485-60c50d772754-cabundle-cert\") pod \"ba7ed3a8-60b0-43fb-8485-60c50d772754\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") "
Oct 30 19:31:21.879259 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.879240    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba7ed3a8-60b0-43fb-8485-60c50d772754-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba7ed3a8-60b0-43fb-8485-60c50d772754-kube-rbac-proxy-sar-config\") pod \"ba7ed3a8-60b0-43fb-8485-60c50d772754\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") "
Oct 30 19:31:21.879307 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.879290    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba7ed3a8-60b0-43fb-8485-60c50d772754-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba7ed3a8-60b0-43fb-8485-60c50d772754-proxy-tls\") pod \"ba7ed3a8-60b0-43fb-8485-60c50d772754\" (UID: \"ba7ed3a8-60b0-43fb-8485-60c50d772754\") "
Oct 30 19:31:21.879498 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.879456    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7ed3a8-60b0-43fb-8485-60c50d772754-kserve-provision-location" pod "ba7ed3a8-60b0-43fb-8485-60c50d772754" (UID: "ba7ed3a8-60b0-43fb-8485-60c50d772754"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:21.879624 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.879566    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7ed3a8-60b0-43fb-8485-60c50d772754-kube-rbac-proxy-sar-config" pod "ba7ed3a8-60b0-43fb-8485-60c50d772754" (UID: "ba7ed3a8-60b0-43fb-8485-60c50d772754"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:21.879624 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.879579    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7ed3a8-60b0-43fb-8485-60c50d772754-cabundle-cert" pod "ba7ed3a8-60b0-43fb-8485-60c50d772754" (UID: "ba7ed3a8-60b0-43fb-8485-60c50d772754"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:21.881237 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.881221    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7ed3a8-60b0-43fb-8485-60c50d772754-kube-api-access-8fv7q" pod "ba7ed3a8-60b0-43fb-8485-60c50d772754" (UID: "ba7ed3a8-60b0-43fb-8485-60c50d772754"). InnerVolumeSpecName "kube-api-access-8fv7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:21.881402 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.881384    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba7ed3a8-60b0-43fb-8485-60c50d772754-proxy-tls" pod "ba7ed3a8-60b0-43fb-8485-60c50d772754" (UID: "ba7ed3a8-60b0-43fb-8485-60c50d772754"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:21.901632 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.901601    4511 generic.go:356] "Generic (PLEG): container finished" podID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerID="ff3e0846b7eb9963ecd0a847bd9a18545306b49aee49ddb9e7d3707d1a4e8f6a" exitCode=0
Oct 30 19:31:21.901730 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.901657    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"
Oct 30 19:31:21.901730 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.901696    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" event={"ID":"ba7ed3a8-60b0-43fb-8485-60c50d772754","Type":"ContainerDied","Data":"ff3e0846b7eb9963ecd0a847bd9a18545306b49aee49ddb9e7d3707d1a4e8f6a"}
Oct 30 19:31:21.901820 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.901755    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7" event={"ID":"ba7ed3a8-60b0-43fb-8485-60c50d772754","Type":"ContainerDied","Data":"171baa23114ede257863b4d59bbc4f084e4ff5d696342c612e246e0c339eb847"}
Oct 30 19:31:21.901820 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.901791    4511 scope.go:122] "RemoveContainer" containerID="d6fcd4aa7f81359515b844d90b71fcba5d57fdd84ff4d9461089b8478f0f0ca0"
Oct 30 19:31:21.909637 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.909623    4511 scope.go:122] "RemoveContainer" containerID="ff3e0846b7eb9963ecd0a847bd9a18545306b49aee49ddb9e7d3707d1a4e8f6a"
Oct 30 19:31:21.917114 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.917100    4511 scope.go:122] "RemoveContainer" containerID="4798ca25c5c05fe40e21e542267aaa9cd6e5ed1c6e5fbaef905e17588b047f0c"
Oct 30 19:31:21.924512 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.924498    4511 scope.go:122] "RemoveContainer" containerID="d6fcd4aa7f81359515b844d90b71fcba5d57fdd84ff4d9461089b8478f0f0ca0"
Oct 30 19:31:21.924774 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:31:21.924757    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6fcd4aa7f81359515b844d90b71fcba5d57fdd84ff4d9461089b8478f0f0ca0\": container with ID starting with d6fcd4aa7f81359515b844d90b71fcba5d57fdd84ff4d9461089b8478f0f0ca0 not found: ID does not exist" containerID="d6fcd4aa7f81359515b844d90b71fcba5d57fdd84ff4d9461089b8478f0f0ca0"
Oct 30 19:31:21.924821 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.924785    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6fcd4aa7f81359515b844d90b71fcba5d57fdd84ff4d9461089b8478f0f0ca0"} err="failed to get container status \"d6fcd4aa7f81359515b844d90b71fcba5d57fdd84ff4d9461089b8478f0f0ca0\": rpc error: code = NotFound desc = could not find container \"d6fcd4aa7f81359515b844d90b71fcba5d57fdd84ff4d9461089b8478f0f0ca0\": container with ID starting with d6fcd4aa7f81359515b844d90b71fcba5d57fdd84ff4d9461089b8478f0f0ca0 not found: ID does not exist"
Oct 30 19:31:21.924821 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.924804    4511 scope.go:122] "RemoveContainer" containerID="ff3e0846b7eb9963ecd0a847bd9a18545306b49aee49ddb9e7d3707d1a4e8f6a"
Oct 30 19:31:21.925041 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:31:21.925021    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3e0846b7eb9963ecd0a847bd9a18545306b49aee49ddb9e7d3707d1a4e8f6a\": container with ID starting with ff3e0846b7eb9963ecd0a847bd9a18545306b49aee49ddb9e7d3707d1a4e8f6a not found: ID does not exist" containerID="ff3e0846b7eb9963ecd0a847bd9a18545306b49aee49ddb9e7d3707d1a4e8f6a"
Oct 30 19:31:21.925108 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.925053    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3e0846b7eb9963ecd0a847bd9a18545306b49aee49ddb9e7d3707d1a4e8f6a"} err="failed to get container status \"ff3e0846b7eb9963ecd0a847bd9a18545306b49aee49ddb9e7d3707d1a4e8f6a\": rpc error: code = NotFound desc = could not find container \"ff3e0846b7eb9963ecd0a847bd9a18545306b49aee49ddb9e7d3707d1a4e8f6a\": container with ID starting with ff3e0846b7eb9963ecd0a847bd9a18545306b49aee49ddb9e7d3707d1a4e8f6a not found: ID does not exist"
Oct 30 19:31:21.925108 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.925078    4511 scope.go:122] "RemoveContainer" containerID="4798ca25c5c05fe40e21e542267aaa9cd6e5ed1c6e5fbaef905e17588b047f0c"
Oct 30 19:31:21.925344 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:31:21.925329    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4798ca25c5c05fe40e21e542267aaa9cd6e5ed1c6e5fbaef905e17588b047f0c\": container with ID starting with 4798ca25c5c05fe40e21e542267aaa9cd6e5ed1c6e5fbaef905e17588b047f0c not found: ID does not exist" containerID="4798ca25c5c05fe40e21e542267aaa9cd6e5ed1c6e5fbaef905e17588b047f0c"
Oct 30 19:31:21.925386 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.925351    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4798ca25c5c05fe40e21e542267aaa9cd6e5ed1c6e5fbaef905e17588b047f0c"} err="failed to get container status \"4798ca25c5c05fe40e21e542267aaa9cd6e5ed1c6e5fbaef905e17588b047f0c\": rpc error: code = NotFound desc = could not find container \"4798ca25c5c05fe40e21e542267aaa9cd6e5ed1c6e5fbaef905e17588b047f0c\": container with ID starting with 4798ca25c5c05fe40e21e542267aaa9cd6e5ed1c6e5fbaef905e17588b047f0c not found: ID does not exist"
Oct 30 19:31:21.979899 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.979874    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba7ed3a8-60b0-43fb-8485-60c50d772754-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:21.979899 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.979898    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba7ed3a8-60b0-43fb-8485-60c50d772754-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:21.980026 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.979908    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba7ed3a8-60b0-43fb-8485-60c50d772754-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:21.980026 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.979917    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fv7q\" (UniqueName: \"kubernetes.io/projected/ba7ed3a8-60b0-43fb-8485-60c50d772754-kube-api-access-8fv7q\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:21.980026 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:21.979926    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba7ed3a8-60b0-43fb-8485-60c50d772754-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:22.168192 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:22.168086    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"]
Oct 30 19:31:22.173659 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:22.173632    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-custom-pass-predictor-777dbdc9bb-rhxl7"]
Oct 30 19:31:23.531100 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:23.531065    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"]
Oct 30 19:31:23.531657 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:23.531350    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kserve-container" containerID="cri-o://3290f9f463a8ac1cb166b92011f484c0ec3e3547c5009383005eb33862a065e5" gracePeriod=30
Oct 30 19:31:23.531657 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:23.531372    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kube-rbac-proxy" containerID="cri-o://48b4584d8621c5765b70db22d9927848987d8096f319179ffe286ba534f3e1a9" gracePeriod=30
Oct 30 19:31:23.655265 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:23.655236    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" path="/var/lib/kubelet/pods/ba7ed3a8-60b0-43fb-8485-60c50d772754/volumes"
Oct 30 19:31:23.912649 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:23.912619    4511 generic.go:356] "Generic (PLEG): container finished" podID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerID="48b4584d8621c5765b70db22d9927848987d8096f319179ffe286ba534f3e1a9" exitCode=0
Oct 30 19:31:23.912845 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:23.912655    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" event={"ID":"42130919-0610-4230-9e13-cf4fb6ec1ec3","Type":"ContainerDied","Data":"48b4584d8621c5765b70db22d9927848987d8096f319179ffe286ba534f3e1a9"}
Oct 30 19:31:27.483767 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:27.483731    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Oct 30 19:31:28.278494 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.278464    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:31:28.330844 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.330817    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42130919-0610-4230-9e13-cf4fb6ec1ec3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42130919-0610-4230-9e13-cf4fb6ec1ec3-cabundle-cert\") pod \"42130919-0610-4230-9e13-cf4fb6ec1ec3\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") "
Oct 30 19:31:28.330964 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.330852    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42130919-0610-4230-9e13-cf4fb6ec1ec3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42130919-0610-4230-9e13-cf4fb6ec1ec3-kube-rbac-proxy-sar-config\") pod \"42130919-0610-4230-9e13-cf4fb6ec1ec3\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") "
Oct 30 19:31:28.330964 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.330888    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42130919-0610-4230-9e13-cf4fb6ec1ec3-proxy-tls\" (UniqueName: \"kubernetes.io/secret/42130919-0610-4230-9e13-cf4fb6ec1ec3-proxy-tls\") pod \"42130919-0610-4230-9e13-cf4fb6ec1ec3\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") "
Oct 30 19:31:28.330964 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.330916    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42130919-0610-4230-9e13-cf4fb6ec1ec3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42130919-0610-4230-9e13-cf4fb6ec1ec3-kserve-provision-location\") pod \"42130919-0610-4230-9e13-cf4fb6ec1ec3\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") "
Oct 30 19:31:28.330964 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.330941    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42130919-0610-4230-9e13-cf4fb6ec1ec3-kube-api-access-hvvsc\" (UniqueName: \"kubernetes.io/projected/42130919-0610-4230-9e13-cf4fb6ec1ec3-kube-api-access-hvvsc\") pod \"42130919-0610-4230-9e13-cf4fb6ec1ec3\" (UID: \"42130919-0610-4230-9e13-cf4fb6ec1ec3\") "
Oct 30 19:31:28.331273 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.331245    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42130919-0610-4230-9e13-cf4fb6ec1ec3-cabundle-cert" pod "42130919-0610-4230-9e13-cf4fb6ec1ec3" (UID: "42130919-0610-4230-9e13-cf4fb6ec1ec3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:28.331339 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.331254    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42130919-0610-4230-9e13-cf4fb6ec1ec3-kube-rbac-proxy-sar-config" pod "42130919-0610-4230-9e13-cf4fb6ec1ec3" (UID: "42130919-0610-4230-9e13-cf4fb6ec1ec3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:28.331339 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.331297    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42130919-0610-4230-9e13-cf4fb6ec1ec3-kserve-provision-location" pod "42130919-0610-4230-9e13-cf4fb6ec1ec3" (UID: "42130919-0610-4230-9e13-cf4fb6ec1ec3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:28.333150 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.333109    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42130919-0610-4230-9e13-cf4fb6ec1ec3-proxy-tls" pod "42130919-0610-4230-9e13-cf4fb6ec1ec3" (UID: "42130919-0610-4230-9e13-cf4fb6ec1ec3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:28.333340 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.333313    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42130919-0610-4230-9e13-cf4fb6ec1ec3-kube-api-access-hvvsc" pod "42130919-0610-4230-9e13-cf4fb6ec1ec3" (UID: "42130919-0610-4230-9e13-cf4fb6ec1ec3"). InnerVolumeSpecName "kube-api-access-hvvsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:28.431910 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.431840    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42130919-0610-4230-9e13-cf4fb6ec1ec3-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:28.431910 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.431862    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42130919-0610-4230-9e13-cf4fb6ec1ec3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:28.431910 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.431875    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42130919-0610-4230-9e13-cf4fb6ec1ec3-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:28.431910 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.431883    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42130919-0610-4230-9e13-cf4fb6ec1ec3-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:28.431910 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.431894    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvvsc\" (UniqueName: \"kubernetes.io/projected/42130919-0610-4230-9e13-cf4fb6ec1ec3-kube-api-access-hvvsc\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:28.934220 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.934190    4511 generic.go:356] "Generic (PLEG): container finished" podID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerID="3290f9f463a8ac1cb166b92011f484c0ec3e3547c5009383005eb33862a065e5" exitCode=0
Oct 30 19:31:28.934220 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.934224    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" event={"ID":"42130919-0610-4230-9e13-cf4fb6ec1ec3","Type":"ContainerDied","Data":"3290f9f463a8ac1cb166b92011f484c0ec3e3547c5009383005eb33862a065e5"}
Oct 30 19:31:28.934844 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.934254    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"
Oct 30 19:31:28.934844 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.934271    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7" event={"ID":"42130919-0610-4230-9e13-cf4fb6ec1ec3","Type":"ContainerDied","Data":"58ea6bcc376b655ea34f8028ae02dec4dd1fcf0cffa1aeb818940b16d4c85788"}
Oct 30 19:31:28.934844 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.934296    4511 scope.go:122] "RemoveContainer" containerID="48b4584d8621c5765b70db22d9927848987d8096f319179ffe286ba534f3e1a9"
Oct 30 19:31:28.943701 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.943674    4511 scope.go:122] "RemoveContainer" containerID="3290f9f463a8ac1cb166b92011f484c0ec3e3547c5009383005eb33862a065e5"
Oct 30 19:31:28.955535 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.955503    4511 scope.go:122] "RemoveContainer" containerID="53243cf7a939021235b5110397c618822d07967206b18c08532d3b00878ccc0e"
Oct 30 19:31:28.963692 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.963674    4511 scope.go:122] "RemoveContainer" containerID="48b4584d8621c5765b70db22d9927848987d8096f319179ffe286ba534f3e1a9"
Oct 30 19:31:28.963954 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:31:28.963931    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48b4584d8621c5765b70db22d9927848987d8096f319179ffe286ba534f3e1a9\": container with ID starting with 48b4584d8621c5765b70db22d9927848987d8096f319179ffe286ba534f3e1a9 not found: ID does not exist" containerID="48b4584d8621c5765b70db22d9927848987d8096f319179ffe286ba534f3e1a9"
Oct 30 19:31:28.964002 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.963967    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b4584d8621c5765b70db22d9927848987d8096f319179ffe286ba534f3e1a9"} err="failed to get container status \"48b4584d8621c5765b70db22d9927848987d8096f319179ffe286ba534f3e1a9\": rpc error: code = NotFound desc = could not find container \"48b4584d8621c5765b70db22d9927848987d8096f319179ffe286ba534f3e1a9\": container with ID starting with 48b4584d8621c5765b70db22d9927848987d8096f319179ffe286ba534f3e1a9 not found: ID does not exist"
Oct 30 19:31:28.964002 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.963988    4511 scope.go:122] "RemoveContainer" containerID="3290f9f463a8ac1cb166b92011f484c0ec3e3547c5009383005eb33862a065e5"
Oct 30 19:31:28.964314 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:31:28.964297    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3290f9f463a8ac1cb166b92011f484c0ec3e3547c5009383005eb33862a065e5\": container with ID starting with 3290f9f463a8ac1cb166b92011f484c0ec3e3547c5009383005eb33862a065e5 not found: ID does not exist" containerID="3290f9f463a8ac1cb166b92011f484c0ec3e3547c5009383005eb33862a065e5"
Oct 30 19:31:28.964360 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.964319    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3290f9f463a8ac1cb166b92011f484c0ec3e3547c5009383005eb33862a065e5"} err="failed to get container status \"3290f9f463a8ac1cb166b92011f484c0ec3e3547c5009383005eb33862a065e5\": rpc error: code = NotFound desc = could not find container \"3290f9f463a8ac1cb166b92011f484c0ec3e3547c5009383005eb33862a065e5\": container with ID starting with 3290f9f463a8ac1cb166b92011f484c0ec3e3547c5009383005eb33862a065e5 not found: ID does not exist"
Oct 30 19:31:28.964360 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.964333    4511 scope.go:122] "RemoveContainer" containerID="53243cf7a939021235b5110397c618822d07967206b18c08532d3b00878ccc0e"
Oct 30 19:31:28.964549 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:31:28.964533    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53243cf7a939021235b5110397c618822d07967206b18c08532d3b00878ccc0e\": container with ID starting with 53243cf7a939021235b5110397c618822d07967206b18c08532d3b00878ccc0e not found: ID does not exist" containerID="53243cf7a939021235b5110397c618822d07967206b18c08532d3b00878ccc0e"
Oct 30 19:31:28.964590 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:28.964554    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53243cf7a939021235b5110397c618822d07967206b18c08532d3b00878ccc0e"} err="failed to get container status \"53243cf7a939021235b5110397c618822d07967206b18c08532d3b00878ccc0e\": rpc error: code = NotFound desc = could not find container \"53243cf7a939021235b5110397c618822d07967206b18c08532d3b00878ccc0e\": container with ID starting with 53243cf7a939021235b5110397c618822d07967206b18c08532d3b00878ccc0e not found: ID does not exist"
Oct 30 19:31:29.092778 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:29.092744    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"]
Oct 30 19:31:29.140435 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:29.140407    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-predictor-575fb6bc48-gk9w7"]
Oct 30 19:31:29.656173 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:29.656121    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" path="/var/lib/kubelet/pods/42130919-0610-4230-9e13-cf4fb6ec1ec3/volumes"
Oct 30 19:31:31.823231 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.823197    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"]
Oct 30 19:31:31.823612 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.823516    4511 cpu_manager.go:414] "containerMap: removing container" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="storage-initializer"
Oct 30 19:31:31.823612 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.823529    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="storage-initializer"
Oct 30 19:31:31.823612 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.823542    4511 cpu_manager.go:414] "containerMap: removing container" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="storage-initializer"
Oct 30 19:31:31.823612 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.823547    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="storage-initializer"
Oct 30 19:31:31.823612 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.823554    4511 cpu_manager.go:414] "containerMap: removing container" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kserve-container"
Oct 30 19:31:31.823612 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.823559    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kserve-container"
Oct 30 19:31:31.823612 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.823567    4511 cpu_manager.go:414] "containerMap: removing container" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kube-rbac-proxy"
Oct 30 19:31:31.823612 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.823571    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kube-rbac-proxy"
Oct 30 19:31:31.823612 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.823577    4511 cpu_manager.go:414] "containerMap: removing container" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kube-rbac-proxy"
Oct 30 19:31:31.823612 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.823584    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kube-rbac-proxy"
Oct 30 19:31:31.823612 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.823602    4511 cpu_manager.go:414] "containerMap: removing container" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kserve-container"
Oct 30 19:31:31.823612 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.823607    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kserve-container"
Oct 30 19:31:31.823964 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.823652    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kserve-container"
Oct 30 19:31:31.823964 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.823658    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="42130919-0610-4230-9e13-cf4fb6ec1ec3" containerName="kube-rbac-proxy"
Oct 30 19:31:31.823964 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.823664    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kube-rbac-proxy"
Oct 30 19:31:31.823964 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.823674    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7ed3a8-60b0-43fb-8485-60c50d772754" containerName="kserve-container"
Oct 30 19:31:31.828471 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.828450    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:31.926364 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.926323    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:31:31.926364 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.926358    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:31:31.926673 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.926358    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:31:31.926673 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.926423    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:31:31.926673 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.926431    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Oct 30 19:31:31.926673 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.926470    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:31:31.926673 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.926568    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:31:31.928177 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.928160    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"]
Oct 30 19:31:31.961927 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.961904    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6046505-56fd-4b40-ad43-97f0299fbdb8-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-57f48d48d4-l594q\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:31.962077 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.961946    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6046505-56fd-4b40-ad43-97f0299fbdb8-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-57f48d48d4-l594q\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:31.962077 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.961982    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6046505-56fd-4b40-ad43-97f0299fbdb8-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-57f48d48d4-l594q\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:31.962240 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.962085    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dl7s\" (UniqueName: \"kubernetes.io/projected/b6046505-56fd-4b40-ad43-97f0299fbdb8-kube-api-access-2dl7s\") pod \"isvc-lightgbm-runtime-predictor-57f48d48d4-l594q\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:31.962240 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:31.962150    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6046505-56fd-4b40-ad43-97f0299fbdb8-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-57f48d48d4-l594q\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:32.062854 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:32.062822    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6046505-56fd-4b40-ad43-97f0299fbdb8-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-57f48d48d4-l594q\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:32.062969 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:32.062869    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6046505-56fd-4b40-ad43-97f0299fbdb8-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-57f48d48d4-l594q\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:32.062969 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:32.062900    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6046505-56fd-4b40-ad43-97f0299fbdb8-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-57f48d48d4-l594q\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:32.062969 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:32.062928    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dl7s\" (UniqueName: \"kubernetes.io/projected/b6046505-56fd-4b40-ad43-97f0299fbdb8-kube-api-access-2dl7s\") pod \"isvc-lightgbm-runtime-predictor-57f48d48d4-l594q\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:32.062969 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:32.062947    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6046505-56fd-4b40-ad43-97f0299fbdb8-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-57f48d48d4-l594q\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:32.063349 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:32.063329    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6046505-56fd-4b40-ad43-97f0299fbdb8-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-57f48d48d4-l594q\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:32.063610 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:32.063590    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6046505-56fd-4b40-ad43-97f0299fbdb8-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-57f48d48d4-l594q\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:32.063675 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:32.063608    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6046505-56fd-4b40-ad43-97f0299fbdb8-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-57f48d48d4-l594q\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:32.065525 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:32.065509    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6046505-56fd-4b40-ad43-97f0299fbdb8-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-57f48d48d4-l594q\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:32.122646 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:32.122593    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dl7s\" (UniqueName: \"kubernetes.io/projected/b6046505-56fd-4b40-ad43-97f0299fbdb8-kube-api-access-2dl7s\") pod \"isvc-lightgbm-runtime-predictor-57f48d48d4-l594q\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:32.145448 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:32.145414    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:32.568028 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:32.567993    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"]
Oct 30 19:31:32.574604 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:31:32.574578    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6046505_56fd_4b40_ad43_97f0299fbdb8.slice/crio-997d94ba0325beec68a5d630a4682c41fa5a0aab5d3f2f3af6ee18ede08eac06 WatchSource:0}: Error finding container 997d94ba0325beec68a5d630a4682c41fa5a0aab5d3f2f3af6ee18ede08eac06: Status 404 returned error can't find the container with id 997d94ba0325beec68a5d630a4682c41fa5a0aab5d3f2f3af6ee18ede08eac06
Oct 30 19:31:32.951817 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:32.951733    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" event={"ID":"b6046505-56fd-4b40-ad43-97f0299fbdb8","Type":"ContainerStarted","Data":"093d18a3d26647fb651ae88a4c007dd33f9b4bfde7450c71ff67230600c99eff"}
Oct 30 19:31:32.951817 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:32.951806    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" event={"ID":"b6046505-56fd-4b40-ad43-97f0299fbdb8","Type":"ContainerStarted","Data":"997d94ba0325beec68a5d630a4682c41fa5a0aab5d3f2f3af6ee18ede08eac06"}
Oct 30 19:31:33.790737 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.790706    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"]
Oct 30 19:31:33.794258 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.794235    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:33.844606 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.844582    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Oct 30 19:31:33.844782 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.844761    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"storage-config\""
Oct 30 19:31:33.844782 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.844777    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:31:33.844876 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.844767    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:31:33.844876 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.844799    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:31:33.877338 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.877317    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf96bd0-75bd-416d-ad5f-0173195e190b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:33.877456 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.877360    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf96bd0-75bd-416d-ad5f-0173195e190b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:33.877456 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.877380    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v4nf\" (UniqueName: \"kubernetes.io/projected/8bf96bd0-75bd-416d-ad5f-0173195e190b-kube-api-access-5v4nf\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:33.877456 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.877441    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf96bd0-75bd-416d-ad5f-0173195e190b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:33.877589 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.877474    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf96bd0-75bd-416d-ad5f-0173195e190b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:33.892061 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.892041    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Oct 30 19:31:33.892306 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.892294    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:31:33.893540 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.893524    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"]
Oct 30 19:31:33.978292 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.978268    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf96bd0-75bd-416d-ad5f-0173195e190b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:33.978622 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.978296    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v4nf\" (UniqueName: \"kubernetes.io/projected/8bf96bd0-75bd-416d-ad5f-0173195e190b-kube-api-access-5v4nf\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:33.978622 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.978316    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf96bd0-75bd-416d-ad5f-0173195e190b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:33.978622 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.978333    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf96bd0-75bd-416d-ad5f-0173195e190b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:33.978622 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.978391    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf96bd0-75bd-416d-ad5f-0173195e190b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:33.978911 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.978688    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf96bd0-75bd-416d-ad5f-0173195e190b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:33.979015 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.978995    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf96bd0-75bd-416d-ad5f-0173195e190b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:33.979172 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.979121    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf96bd0-75bd-416d-ad5f-0173195e190b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:33.981007 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:33.980983    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf96bd0-75bd-416d-ad5f-0173195e190b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:34.037542 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:34.037514    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v4nf\" (UniqueName: \"kubernetes.io/projected/8bf96bd0-75bd-416d-ad5f-0173195e190b-kube-api-access-5v4nf\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") " pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:34.111652 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:34.111629    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:34.532271 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:34.532191    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"]
Oct 30 19:31:34.538762 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:31:34.538729    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf96bd0_75bd_416d_ad5f_0173195e190b.slice/crio-79cc02b45f8f7f6c3b1039d5b86f1a2c6eae992e1c4984644225ebba666a2aac WatchSource:0}: Error finding container 79cc02b45f8f7f6c3b1039d5b86f1a2c6eae992e1c4984644225ebba666a2aac: Status 404 returned error can't find the container with id 79cc02b45f8f7f6c3b1039d5b86f1a2c6eae992e1c4984644225ebba666a2aac
Oct 30 19:31:34.962839 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:34.962802    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" event={"ID":"8bf96bd0-75bd-416d-ad5f-0173195e190b","Type":"ContainerStarted","Data":"a315cef1791cc2bb05af9664c33e163d88e2e548beb337e418a52f746c5b31cf"}
Oct 30 19:31:34.963007 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:34.962860    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" event={"ID":"8bf96bd0-75bd-416d-ad5f-0173195e190b","Type":"ContainerStarted","Data":"79cc02b45f8f7f6c3b1039d5b86f1a2c6eae992e1c4984644225ebba666a2aac"}
Oct 30 19:31:35.968475 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:35.968398    4511 generic.go:356] "Generic (PLEG): container finished" podID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerID="a315cef1791cc2bb05af9664c33e163d88e2e548beb337e418a52f746c5b31cf" exitCode=0
Oct 30 19:31:35.968858 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:35.968475    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" event={"ID":"8bf96bd0-75bd-416d-ad5f-0173195e190b","Type":"ContainerDied","Data":"a315cef1791cc2bb05af9664c33e163d88e2e548beb337e418a52f746c5b31cf"}
Oct 30 19:31:36.973793 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:36.973757    4511 generic.go:356] "Generic (PLEG): container finished" podID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerID="093d18a3d26647fb651ae88a4c007dd33f9b4bfde7450c71ff67230600c99eff" exitCode=0
Oct 30 19:31:36.974242 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:36.973831    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" event={"ID":"b6046505-56fd-4b40-ad43-97f0299fbdb8","Type":"ContainerDied","Data":"093d18a3d26647fb651ae88a4c007dd33f9b4bfde7450c71ff67230600c99eff"}
Oct 30 19:31:36.976652 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:36.976626    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" event={"ID":"8bf96bd0-75bd-416d-ad5f-0173195e190b","Type":"ContainerStarted","Data":"d93d60b82fe7994e477798528c40a48c6fb58c8001181618b7d024c6125de7f2"}
Oct 30 19:31:36.976764 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:36.976682    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" event={"ID":"8bf96bd0-75bd-416d-ad5f-0173195e190b","Type":"ContainerStarted","Data":"540c0e573d1b62766018652ccc37c06c2d67346ae6518673407c20bffa96b19e"}
Oct 30 19:31:36.976859 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:36.976824    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:36.976894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:36.976871    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:36.978345 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:36.978321    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 30 19:31:37.205855 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:37.205802    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" podStartSLOduration=3.209934705 podStartE2EDuration="4.205789115s" totalImagesPullingTime="0s" totalInitContainerRuntime="995.85441ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:37.204755002 +0000 UTC m=+3191.722439902" watchObservedRunningTime="2026-10-30 19:31:37.205789115 +0000 UTC m=+3191.723474062"
Oct 30 19:31:37.982667 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:37.982634    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" event={"ID":"b6046505-56fd-4b40-ad43-97f0299fbdb8","Type":"ContainerStarted","Data":"eee905d29acdab971bbb6f10cccc3c35743cbccc696d1d13c5b6750d6c501149"}
Oct 30 19:31:37.983075 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:37.982700    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" event={"ID":"b6046505-56fd-4b40-ad43-97f0299fbdb8","Type":"ContainerStarted","Data":"5407f9d4c01f5c4c5c987d7457ac8983c95b87b9b5f5169932772260d01a71da"}
Oct 30 19:31:37.983075 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:37.983014    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 30 19:31:37.983185 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:37.983080    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:38.098860 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:38.098804    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" podStartSLOduration=3.371612843 podStartE2EDuration="7.098790172s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.727177329s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:38.097458853 +0000 UTC m=+3192.615143753" watchObservedRunningTime="2026-10-30 19:31:38.098790172 +0000 UTC m=+3192.616475071"
Oct 30 19:31:38.987067 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:38.987032    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:38.988364 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:38.988338    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 30 19:31:39.991424 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:39.991382    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 30 19:31:40.156343 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:31:40.156305    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:31:42.987849 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:42.987819    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:31:42.988313 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:42.988288    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 30 19:31:44.996065 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:44.996035    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:31:44.996593 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:44.996560    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 30 19:31:52.988483 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:52.988447    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 30 19:31:54.997316 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:31:54.997275    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 30 19:32:02.989115 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:32:02.989074    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 30 19:32:04.997033 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:32:04.996980    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 30 19:32:10.157209 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:32:10.157174    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:32:12.988750 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:32:12.988707    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 30 19:32:14.996592 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:32:14.996554    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 30 19:32:22.989068 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:32:22.989029    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 30 19:32:24.996533 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:32:24.996492    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 30 19:32:32.988882 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:32:32.988841    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 30 19:32:34.997346 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:32:34.997307    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 30 19:32:40.156055 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:32:40.156016    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:32:42.988300 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:32:42.988263    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 30 19:32:44.996578 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:32:44.996531    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 30 19:32:46.651783 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:32:46.651737    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 30 19:32:52.989293 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:32:52.989262    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:32:56.652046 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:32:56.652006    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 30 19:33:02.317568 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:02.317531    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"]
Oct 30 19:33:02.318080 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:02.317898    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kserve-container" containerID="cri-o://540c0e573d1b62766018652ccc37c06c2d67346ae6518673407c20bffa96b19e" gracePeriod=30
Oct 30 19:33:02.318080 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:02.317906    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kube-rbac-proxy" containerID="cri-o://d93d60b82fe7994e477798528c40a48c6fb58c8001181618b7d024c6125de7f2" gracePeriod=30
Oct 30 19:33:02.984182 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:02.984117    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Oct 30 19:33:02.988459 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:02.988436    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Oct 30 19:33:03.318216 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:03.318181    4511 generic.go:356] "Generic (PLEG): container finished" podID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerID="d93d60b82fe7994e477798528c40a48c6fb58c8001181618b7d024c6125de7f2" exitCode=0
Oct 30 19:33:03.318693 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:03.318250    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" event={"ID":"8bf96bd0-75bd-416d-ad5f-0173195e190b","Type":"ContainerDied","Data":"d93d60b82fe7994e477798528c40a48c6fb58c8001181618b7d024c6125de7f2"}
Oct 30 19:33:06.652385 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:06.652304    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:33:06.875640 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:06.875613    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:33:06.896750 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:06.896722    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf96bd0-75bd-416d-ad5f-0173195e190b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf96bd0-75bd-416d-ad5f-0173195e190b-kserve-provision-location\") pod \"8bf96bd0-75bd-416d-ad5f-0173195e190b\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") "
Oct 30 19:33:06.896895 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:06.896762    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bf96bd0-75bd-416d-ad5f-0173195e190b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf96bd0-75bd-416d-ad5f-0173195e190b-proxy-tls\") pod \"8bf96bd0-75bd-416d-ad5f-0173195e190b\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") "
Oct 30 19:33:06.896895 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:06.896792    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf96bd0-75bd-416d-ad5f-0173195e190b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf96bd0-75bd-416d-ad5f-0173195e190b-cabundle-cert\") pod \"8bf96bd0-75bd-416d-ad5f-0173195e190b\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") "
Oct 30 19:33:06.896895 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:06.896866    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf96bd0-75bd-416d-ad5f-0173195e190b-kube-api-access-5v4nf\" (UniqueName: \"kubernetes.io/projected/8bf96bd0-75bd-416d-ad5f-0173195e190b-kube-api-access-5v4nf\") pod \"8bf96bd0-75bd-416d-ad5f-0173195e190b\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") "
Oct 30 19:33:06.897070 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:06.896909    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf96bd0-75bd-416d-ad5f-0173195e190b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf96bd0-75bd-416d-ad5f-0173195e190b-kube-rbac-proxy-sar-config\") pod \"8bf96bd0-75bd-416d-ad5f-0173195e190b\" (UID: \"8bf96bd0-75bd-416d-ad5f-0173195e190b\") "
Oct 30 19:33:06.897070 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:06.897036    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf96bd0-75bd-416d-ad5f-0173195e190b-kserve-provision-location" pod "8bf96bd0-75bd-416d-ad5f-0173195e190b" (UID: "8bf96bd0-75bd-416d-ad5f-0173195e190b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:06.897205 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:06.897161    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf96bd0-75bd-416d-ad5f-0173195e190b-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:06.897306 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:06.897279    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf96bd0-75bd-416d-ad5f-0173195e190b-cabundle-cert" pod "8bf96bd0-75bd-416d-ad5f-0173195e190b" (UID: "8bf96bd0-75bd-416d-ad5f-0173195e190b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:06.897373 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:06.897311    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf96bd0-75bd-416d-ad5f-0173195e190b-kube-rbac-proxy-sar-config" pod "8bf96bd0-75bd-416d-ad5f-0173195e190b" (UID: "8bf96bd0-75bd-416d-ad5f-0173195e190b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:06.898981 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:06.898957    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf96bd0-75bd-416d-ad5f-0173195e190b-kube-api-access-5v4nf" pod "8bf96bd0-75bd-416d-ad5f-0173195e190b" (UID: "8bf96bd0-75bd-416d-ad5f-0173195e190b"). InnerVolumeSpecName "kube-api-access-5v4nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:06.899067 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:06.898997    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf96bd0-75bd-416d-ad5f-0173195e190b-proxy-tls" pod "8bf96bd0-75bd-416d-ad5f-0173195e190b" (UID: "8bf96bd0-75bd-416d-ad5f-0173195e190b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:06.997980 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:06.997956    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v4nf\" (UniqueName: \"kubernetes.io/projected/8bf96bd0-75bd-416d-ad5f-0173195e190b-kube-api-access-5v4nf\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:06.997980 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:06.997982    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf96bd0-75bd-416d-ad5f-0173195e190b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:06.998146 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:06.997993    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf96bd0-75bd-416d-ad5f-0173195e190b-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:06.998146 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:06.998003    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf96bd0-75bd-416d-ad5f-0173195e190b-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:07.336709 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:07.336675    4511 generic.go:356] "Generic (PLEG): container finished" podID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerID="540c0e573d1b62766018652ccc37c06c2d67346ae6518673407c20bffa96b19e" exitCode=0
Oct 30 19:33:07.336869 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:07.336729    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"
Oct 30 19:33:07.336869 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:07.336744    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" event={"ID":"8bf96bd0-75bd-416d-ad5f-0173195e190b","Type":"ContainerDied","Data":"540c0e573d1b62766018652ccc37c06c2d67346ae6518673407c20bffa96b19e"}
Oct 30 19:33:07.336869 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:07.336791    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p" event={"ID":"8bf96bd0-75bd-416d-ad5f-0173195e190b","Type":"ContainerDied","Data":"79cc02b45f8f7f6c3b1039d5b86f1a2c6eae992e1c4984644225ebba666a2aac"}
Oct 30 19:33:07.336869 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:07.336816    4511 scope.go:122] "RemoveContainer" containerID="d93d60b82fe7994e477798528c40a48c6fb58c8001181618b7d024c6125de7f2"
Oct 30 19:33:07.345594 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:07.345576    4511 scope.go:122] "RemoveContainer" containerID="540c0e573d1b62766018652ccc37c06c2d67346ae6518673407c20bffa96b19e"
Oct 30 19:33:07.352733 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:07.352716    4511 scope.go:122] "RemoveContainer" containerID="a315cef1791cc2bb05af9664c33e163d88e2e548beb337e418a52f746c5b31cf"
Oct 30 19:33:07.359487 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:07.359473    4511 scope.go:122] "RemoveContainer" containerID="d93d60b82fe7994e477798528c40a48c6fb58c8001181618b7d024c6125de7f2"
Oct 30 19:33:07.359726 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:33:07.359710    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93d60b82fe7994e477798528c40a48c6fb58c8001181618b7d024c6125de7f2\": container with ID starting with d93d60b82fe7994e477798528c40a48c6fb58c8001181618b7d024c6125de7f2 not found: ID does not exist" containerID="d93d60b82fe7994e477798528c40a48c6fb58c8001181618b7d024c6125de7f2"
Oct 30 19:33:07.359793 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:07.359733    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93d60b82fe7994e477798528c40a48c6fb58c8001181618b7d024c6125de7f2"} err="failed to get container status \"d93d60b82fe7994e477798528c40a48c6fb58c8001181618b7d024c6125de7f2\": rpc error: code = NotFound desc = could not find container \"d93d60b82fe7994e477798528c40a48c6fb58c8001181618b7d024c6125de7f2\": container with ID starting with d93d60b82fe7994e477798528c40a48c6fb58c8001181618b7d024c6125de7f2 not found: ID does not exist"
Oct 30 19:33:07.359793 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:07.359750    4511 scope.go:122] "RemoveContainer" containerID="540c0e573d1b62766018652ccc37c06c2d67346ae6518673407c20bffa96b19e"
Oct 30 19:33:07.359960 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:33:07.359940    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"540c0e573d1b62766018652ccc37c06c2d67346ae6518673407c20bffa96b19e\": container with ID starting with 540c0e573d1b62766018652ccc37c06c2d67346ae6518673407c20bffa96b19e not found: ID does not exist" containerID="540c0e573d1b62766018652ccc37c06c2d67346ae6518673407c20bffa96b19e"
Oct 30 19:33:07.360026 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:07.359972    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"540c0e573d1b62766018652ccc37c06c2d67346ae6518673407c20bffa96b19e"} err="failed to get container status \"540c0e573d1b62766018652ccc37c06c2d67346ae6518673407c20bffa96b19e\": rpc error: code = NotFound desc = could not find container \"540c0e573d1b62766018652ccc37c06c2d67346ae6518673407c20bffa96b19e\": container with ID starting with 540c0e573d1b62766018652ccc37c06c2d67346ae6518673407c20bffa96b19e not found: ID does not exist"
Oct 30 19:33:07.360026 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:07.359998    4511 scope.go:122] "RemoveContainer" containerID="a315cef1791cc2bb05af9664c33e163d88e2e548beb337e418a52f746c5b31cf"
Oct 30 19:33:07.360253 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:33:07.360237    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a315cef1791cc2bb05af9664c33e163d88e2e548beb337e418a52f746c5b31cf\": container with ID starting with a315cef1791cc2bb05af9664c33e163d88e2e548beb337e418a52f746c5b31cf not found: ID does not exist" containerID="a315cef1791cc2bb05af9664c33e163d88e2e548beb337e418a52f746c5b31cf"
Oct 30 19:33:07.360303 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:07.360260    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a315cef1791cc2bb05af9664c33e163d88e2e548beb337e418a52f746c5b31cf"} err="failed to get container status \"a315cef1791cc2bb05af9664c33e163d88e2e548beb337e418a52f746c5b31cf\": rpc error: code = NotFound desc = could not find container \"a315cef1791cc2bb05af9664c33e163d88e2e548beb337e418a52f746c5b31cf\": container with ID starting with a315cef1791cc2bb05af9664c33e163d88e2e548beb337e418a52f746c5b31cf not found: ID does not exist"
Oct 30 19:33:07.494565 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:07.494536    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"]
Oct 30 19:33:07.542896 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:07.542871    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-sklearn-s3-tls-serving-pass-predictor-6d8d9d69d7-jmf2p"]
Oct 30 19:33:07.655550 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:07.655480    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" path="/var/lib/kubelet/pods/8bf96bd0-75bd-416d-ad5f-0173195e190b/volumes"
Oct 30 19:33:10.156087 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:33:10.156046    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:33:13.353431 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:13.353402    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"]
Oct 30 19:33:13.353806 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:13.353660    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kserve-container" containerID="cri-o://5407f9d4c01f5c4c5c987d7457ac8983c95b87b9b5f5169932772260d01a71da" gracePeriod=30
Oct 30 19:33:13.353806 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:13.353671    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kube-rbac-proxy" containerID="cri-o://eee905d29acdab971bbb6f10cccc3c35743cbccc696d1d13c5b6750d6c501149" gracePeriod=30
Oct 30 19:33:14.366872 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:14.366839    4511 generic.go:356] "Generic (PLEG): container finished" podID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerID="eee905d29acdab971bbb6f10cccc3c35743cbccc696d1d13c5b6750d6c501149" exitCode=0
Oct 30 19:33:14.367268 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:14.366878    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" event={"ID":"b6046505-56fd-4b40-ad43-97f0299fbdb8","Type":"ContainerDied","Data":"eee905d29acdab971bbb6f10cccc3c35743cbccc696d1d13c5b6750d6c501149"}
Oct 30 19:33:14.991530 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:14.991488    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Oct 30 19:33:16.651658 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:16.651613    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Oct 30 19:33:18.383923 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.383887    4511 generic.go:356] "Generic (PLEG): container finished" podID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerID="5407f9d4c01f5c4c5c987d7457ac8983c95b87b9b5f5169932772260d01a71da" exitCode=0
Oct 30 19:33:18.384296 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.383946    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" event={"ID":"b6046505-56fd-4b40-ad43-97f0299fbdb8","Type":"ContainerDied","Data":"5407f9d4c01f5c4c5c987d7457ac8983c95b87b9b5f5169932772260d01a71da"}
Oct 30 19:33:18.491929 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.491905    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:33:18.577725 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.577698    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6046505-56fd-4b40-ad43-97f0299fbdb8-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6046505-56fd-4b40-ad43-97f0299fbdb8-kserve-provision-location\") pod \"b6046505-56fd-4b40-ad43-97f0299fbdb8\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") "
Oct 30 19:33:18.577871 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.577757    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6046505-56fd-4b40-ad43-97f0299fbdb8-proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6046505-56fd-4b40-ad43-97f0299fbdb8-proxy-tls\") pod \"b6046505-56fd-4b40-ad43-97f0299fbdb8\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") "
Oct 30 19:33:18.577871 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.577790    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6046505-56fd-4b40-ad43-97f0299fbdb8-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6046505-56fd-4b40-ad43-97f0299fbdb8-cabundle-cert\") pod \"b6046505-56fd-4b40-ad43-97f0299fbdb8\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") "
Oct 30 19:33:18.577871 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.577811    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6046505-56fd-4b40-ad43-97f0299fbdb8-kube-api-access-2dl7s\" (UniqueName: \"kubernetes.io/projected/b6046505-56fd-4b40-ad43-97f0299fbdb8-kube-api-access-2dl7s\") pod \"b6046505-56fd-4b40-ad43-97f0299fbdb8\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") "
Oct 30 19:33:18.577871 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.577830    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6046505-56fd-4b40-ad43-97f0299fbdb8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6046505-56fd-4b40-ad43-97f0299fbdb8-kube-rbac-proxy-sar-config\") pod \"b6046505-56fd-4b40-ad43-97f0299fbdb8\" (UID: \"b6046505-56fd-4b40-ad43-97f0299fbdb8\") "
Oct 30 19:33:18.578094 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.578051    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6046505-56fd-4b40-ad43-97f0299fbdb8-kserve-provision-location" pod "b6046505-56fd-4b40-ad43-97f0299fbdb8" (UID: "b6046505-56fd-4b40-ad43-97f0299fbdb8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:18.578177 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.578148    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6046505-56fd-4b40-ad43-97f0299fbdb8-cabundle-cert" pod "b6046505-56fd-4b40-ad43-97f0299fbdb8" (UID: "b6046505-56fd-4b40-ad43-97f0299fbdb8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:18.578295 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.578274    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6046505-56fd-4b40-ad43-97f0299fbdb8-kube-rbac-proxy-sar-config" pod "b6046505-56fd-4b40-ad43-97f0299fbdb8" (UID: "b6046505-56fd-4b40-ad43-97f0299fbdb8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:18.579894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.579867    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6046505-56fd-4b40-ad43-97f0299fbdb8-kube-api-access-2dl7s" pod "b6046505-56fd-4b40-ad43-97f0299fbdb8" (UID: "b6046505-56fd-4b40-ad43-97f0299fbdb8"). InnerVolumeSpecName "kube-api-access-2dl7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:18.579987 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.579937    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6046505-56fd-4b40-ad43-97f0299fbdb8-proxy-tls" pod "b6046505-56fd-4b40-ad43-97f0299fbdb8" (UID: "b6046505-56fd-4b40-ad43-97f0299fbdb8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:18.678639 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.678580    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6046505-56fd-4b40-ad43-97f0299fbdb8-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:18.678639 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.678615    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6046505-56fd-4b40-ad43-97f0299fbdb8-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:18.678639 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.678625    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6046505-56fd-4b40-ad43-97f0299fbdb8-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:18.678639 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.678633    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dl7s\" (UniqueName: \"kubernetes.io/projected/b6046505-56fd-4b40-ad43-97f0299fbdb8-kube-api-access-2dl7s\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:18.678639 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:18.678642    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6046505-56fd-4b40-ad43-97f0299fbdb8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:19.390013 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:19.389976    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q" event={"ID":"b6046505-56fd-4b40-ad43-97f0299fbdb8","Type":"ContainerDied","Data":"997d94ba0325beec68a5d630a4682c41fa5a0aab5d3f2f3af6ee18ede08eac06"}
Oct 30 19:33:19.390013 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:19.390000    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"
Oct 30 19:33:19.390703 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:19.390041    4511 scope.go:122] "RemoveContainer" containerID="eee905d29acdab971bbb6f10cccc3c35743cbccc696d1d13c5b6750d6c501149"
Oct 30 19:33:19.398931 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:19.398911    4511 scope.go:122] "RemoveContainer" containerID="5407f9d4c01f5c4c5c987d7457ac8983c95b87b9b5f5169932772260d01a71da"
Oct 30 19:33:19.405850 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:19.405833    4511 scope.go:122] "RemoveContainer" containerID="093d18a3d26647fb651ae88a4c007dd33f9b4bfde7450c71ff67230600c99eff"
Oct 30 19:33:19.595848 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:19.595822    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"]
Oct 30 19:33:19.643318 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:19.643261    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-runtime-predictor-57f48d48d4-l594q"]
Oct 30 19:33:19.656800 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:19.656775    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" path="/var/lib/kubelet/pods/b6046505-56fd-4b40-ad43-97f0299fbdb8/volumes"
Oct 30 19:33:22.458819 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.458786    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"]
Oct 30 19:33:22.459195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.459079    4511 cpu_manager.go:414] "containerMap: removing container" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="storage-initializer"
Oct 30 19:33:22.459195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.459089    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="storage-initializer"
Oct 30 19:33:22.459195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.459102    4511 cpu_manager.go:414] "containerMap: removing container" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kube-rbac-proxy"
Oct 30 19:33:22.459195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.459107    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kube-rbac-proxy"
Oct 30 19:33:22.459195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.459115    4511 cpu_manager.go:414] "containerMap: removing container" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kserve-container"
Oct 30 19:33:22.459195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.459120    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kserve-container"
Oct 30 19:33:22.459195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.459149    4511 cpu_manager.go:414] "containerMap: removing container" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kserve-container"
Oct 30 19:33:22.459195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.459156    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kserve-container"
Oct 30 19:33:22.459195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.459170    4511 cpu_manager.go:414] "containerMap: removing container" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kube-rbac-proxy"
Oct 30 19:33:22.459195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.459176    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kube-rbac-proxy"
Oct 30 19:33:22.459195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.459190    4511 cpu_manager.go:414] "containerMap: removing container" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="storage-initializer"
Oct 30 19:33:22.459195 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.459194    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="storage-initializer"
Oct 30 19:33:22.459568 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.459238    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kube-rbac-proxy"
Oct 30 19:33:22.459568 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.459246    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kserve-container"
Oct 30 19:33:22.459568 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.459254    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf96bd0-75bd-416d-ad5f-0173195e190b" containerName="kserve-container"
Oct 30 19:33:22.459568 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.459260    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6046505-56fd-4b40-ad43-97f0299fbdb8" containerName="kube-rbac-proxy"
Oct 30 19:33:22.463767 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.463748    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:22.510033 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.510001    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:22.510186 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.510048    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8jk5\" (UniqueName: \"kubernetes.io/projected/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kube-api-access-g8jk5\") pod \"isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:22.510266 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.510199    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:22.510266 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.510235    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:22.510375 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.510272    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:22.561542 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.561516    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:33:22.561675 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.561653    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:33:22.561675 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.561656    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:33:22.561832 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.561671    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:33:22.561832 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.561713    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Oct 30 19:33:22.561832 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.561714    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:33:22.561832 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.561729    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:33:22.565620 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.565601    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"]
Oct 30 19:33:22.610715 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.610692    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8jk5\" (UniqueName: \"kubernetes.io/projected/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kube-api-access-g8jk5\") pod \"isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:22.610840 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.610740    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:22.610840 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.610762    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:22.610840 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.610793    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:22.610840 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.610817    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:22.611328 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.611306    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:22.611524 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.611477    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:22.611524 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.611498    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:22.613279 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.613261    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:22.669661 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.669642    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8jk5\" (UniqueName: \"kubernetes.io/projected/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kube-api-access-g8jk5\") pod \"isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:22.780670 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:22.780616    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:23.202200 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:23.202173    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"]
Oct 30 19:33:23.208798 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:33:23.208770    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d86fc4_170a_43d7_ad3b_3ecbc00357c0.slice/crio-b0a08f15117d21af0824b6bb2d8f06527377a17dd2d24c7625cc04468f6a0100 WatchSource:0}: Error finding container b0a08f15117d21af0824b6bb2d8f06527377a17dd2d24c7625cc04468f6a0100: Status 404 returned error can't find the container with id b0a08f15117d21af0824b6bb2d8f06527377a17dd2d24c7625cc04468f6a0100
Oct 30 19:33:23.406072 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:23.406037    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2" event={"ID":"91d86fc4-170a-43d7-ad3b-3ecbc00357c0","Type":"ContainerStarted","Data":"a2549d7ff01ae27d6c165716f473a6b7993896fa772dc0d06b340ee47694fbfb"}
Oct 30 19:33:23.406251 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:23.406106    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2" event={"ID":"91d86fc4-170a-43d7-ad3b-3ecbc00357c0","Type":"ContainerStarted","Data":"b0a08f15117d21af0824b6bb2d8f06527377a17dd2d24c7625cc04468f6a0100"}
Oct 30 19:33:25.888103 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:25.887985    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 19:33:25.914488 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:25.897687    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 19:33:27.421870 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:27.421836    4511 generic.go:356] "Generic (PLEG): container finished" podID="91d86fc4-170a-43d7-ad3b-3ecbc00357c0" containerID="a2549d7ff01ae27d6c165716f473a6b7993896fa772dc0d06b340ee47694fbfb" exitCode=0
Oct 30 19:33:27.422260 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:27.421884    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2" event={"ID":"91d86fc4-170a-43d7-ad3b-3ecbc00357c0","Type":"ContainerDied","Data":"a2549d7ff01ae27d6c165716f473a6b7993896fa772dc0d06b340ee47694fbfb"}
Oct 30 19:33:28.428357 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:28.428316    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2" event={"ID":"91d86fc4-170a-43d7-ad3b-3ecbc00357c0","Type":"ContainerStarted","Data":"f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027"}
Oct 30 19:33:28.428855 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:28.428386    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2" event={"ID":"91d86fc4-170a-43d7-ad3b-3ecbc00357c0","Type":"ContainerStarted","Data":"1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333"}
Oct 30 19:33:28.428855 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:28.428575    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:28.550143 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:28.550062    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2" podStartSLOduration=2.585799799 podStartE2EDuration="6.550042202s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.964242403s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:28.549516178 +0000 UTC m=+3303.067201081" watchObservedRunningTime="2026-10-30 19:33:28.550042202 +0000 UTC m=+3303.067727105"
Oct 30 19:33:29.432664 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:29.432636    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:35.442060 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:33:35.442024    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:33:40.157389 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:33:40.157354    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:34:05.445755 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:05.445727    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:34:10.156592 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:34:10.156556    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:34:12.643259 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:12.643230    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"]
Oct 30 19:34:12.643653 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:12.643465    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2" podUID="91d86fc4-170a-43d7-ad3b-3ecbc00357c0" containerName="kserve-container" containerID="cri-o://1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333" gracePeriod=30
Oct 30 19:34:12.643653 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:12.643482    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2" podUID="91d86fc4-170a-43d7-ad3b-3ecbc00357c0" containerName="kube-rbac-proxy" containerID="cri-o://f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027" gracePeriod=30
Oct 30 19:34:13.593427 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.593394    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:34:13.596592 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.596567    4511 generic.go:356] "Generic (PLEG): container finished" podID="91d86fc4-170a-43d7-ad3b-3ecbc00357c0" containerID="f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027" exitCode=0
Oct 30 19:34:13.596592 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.596588    4511 generic.go:356] "Generic (PLEG): container finished" podID="91d86fc4-170a-43d7-ad3b-3ecbc00357c0" containerID="1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333" exitCode=0
Oct 30 19:34:13.596731 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.596613    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"
Oct 30 19:34:13.596731 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.596644    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2" event={"ID":"91d86fc4-170a-43d7-ad3b-3ecbc00357c0","Type":"ContainerDied","Data":"f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027"}
Oct 30 19:34:13.596731 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.596704    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2" event={"ID":"91d86fc4-170a-43d7-ad3b-3ecbc00357c0","Type":"ContainerDied","Data":"1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333"}
Oct 30 19:34:13.596827 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.596751    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2" event={"ID":"91d86fc4-170a-43d7-ad3b-3ecbc00357c0","Type":"ContainerDied","Data":"b0a08f15117d21af0824b6bb2d8f06527377a17dd2d24c7625cc04468f6a0100"}
Oct 30 19:34:13.596827 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.596765    4511 scope.go:122] "RemoveContainer" containerID="f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027"
Oct 30 19:34:13.604378 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.604352    4511 scope.go:122] "RemoveContainer" containerID="1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333"
Oct 30 19:34:13.611864 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.611839    4511 scope.go:122] "RemoveContainer" containerID="a2549d7ff01ae27d6c165716f473a6b7993896fa772dc0d06b340ee47694fbfb"
Oct 30 19:34:13.618990 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.618972    4511 scope.go:122] "RemoveContainer" containerID="f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027"
Oct 30 19:34:13.619261 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:34:13.619243    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027\": container with ID starting with f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027 not found: ID does not exist" containerID="f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027"
Oct 30 19:34:13.619335 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.619274    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027"} err="failed to get container status \"f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027\": rpc error: code = NotFound desc = could not find container \"f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027\": container with ID starting with f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027 not found: ID does not exist"
Oct 30 19:34:13.619335 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.619296    4511 scope.go:122] "RemoveContainer" containerID="1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333"
Oct 30 19:34:13.619525 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:34:13.619508    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333\": container with ID starting with 1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333 not found: ID does not exist" containerID="1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333"
Oct 30 19:34:13.619567 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.619532    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333"} err="failed to get container status \"1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333\": rpc error: code = NotFound desc = could not find container \"1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333\": container with ID starting with 1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333 not found: ID does not exist"
Oct 30 19:34:13.619567 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.619545    4511 scope.go:122] "RemoveContainer" containerID="a2549d7ff01ae27d6c165716f473a6b7993896fa772dc0d06b340ee47694fbfb"
Oct 30 19:34:13.619771 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:34:13.619751    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2549d7ff01ae27d6c165716f473a6b7993896fa772dc0d06b340ee47694fbfb\": container with ID starting with a2549d7ff01ae27d6c165716f473a6b7993896fa772dc0d06b340ee47694fbfb not found: ID does not exist" containerID="a2549d7ff01ae27d6c165716f473a6b7993896fa772dc0d06b340ee47694fbfb"
Oct 30 19:34:13.619835 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.619779    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2549d7ff01ae27d6c165716f473a6b7993896fa772dc0d06b340ee47694fbfb"} err="failed to get container status \"a2549d7ff01ae27d6c165716f473a6b7993896fa772dc0d06b340ee47694fbfb\": rpc error: code = NotFound desc = could not find container \"a2549d7ff01ae27d6c165716f473a6b7993896fa772dc0d06b340ee47694fbfb\": container with ID starting with a2549d7ff01ae27d6c165716f473a6b7993896fa772dc0d06b340ee47694fbfb not found: ID does not exist"
Oct 30 19:34:13.619835 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.619797    4511 scope.go:122] "RemoveContainer" containerID="f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027"
Oct 30 19:34:13.620002 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.619983    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027"} err="failed to get container status \"f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027\": rpc error: code = NotFound desc = could not find container \"f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027\": container with ID starting with f8917aa050d62aec776495ff1793b1b63355d4932179199eed1939812d161027 not found: ID does not exist"
Oct 30 19:34:13.620051 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.620005    4511 scope.go:122] "RemoveContainer" containerID="1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333"
Oct 30 19:34:13.620223 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.620205    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333"} err="failed to get container status \"1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333\": rpc error: code = NotFound desc = could not find container \"1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333\": container with ID starting with 1ca213309cfe246bbab031e6a324642c00408f852eb69340ac89f23f48f0e333 not found: ID does not exist"
Oct 30 19:34:13.620299 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.620224    4511 scope.go:122] "RemoveContainer" containerID="a2549d7ff01ae27d6c165716f473a6b7993896fa772dc0d06b340ee47694fbfb"
Oct 30 19:34:13.620474 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.620449    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2549d7ff01ae27d6c165716f473a6b7993896fa772dc0d06b340ee47694fbfb"} err="failed to get container status \"a2549d7ff01ae27d6c165716f473a6b7993896fa772dc0d06b340ee47694fbfb\": rpc error: code = NotFound desc = could not find container \"a2549d7ff01ae27d6c165716f473a6b7993896fa772dc0d06b340ee47694fbfb\": container with ID starting with a2549d7ff01ae27d6c165716f473a6b7993896fa772dc0d06b340ee47694fbfb not found: ID does not exist"
Oct 30 19:34:13.689048 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.689024    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-proxy-tls\" (UniqueName: \"kubernetes.io/secret/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-proxy-tls\") pod \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") "
Oct 30 19:34:13.689459 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.689110    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kserve-provision-location\") pod \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") "
Oct 30 19:34:13.689459 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.689158    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-cabundle-cert\") pod \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") "
Oct 30 19:34:13.689459 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.689191    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kube-rbac-proxy-sar-config\") pod \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") "
Oct 30 19:34:13.689459 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.689218    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kube-api-access-g8jk5\" (UniqueName: \"kubernetes.io/projected/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kube-api-access-g8jk5\") pod \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\" (UID: \"91d86fc4-170a-43d7-ad3b-3ecbc00357c0\") "
Oct 30 19:34:13.689686 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.689473    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kserve-provision-location" pod "91d86fc4-170a-43d7-ad3b-3ecbc00357c0" (UID: "91d86fc4-170a-43d7-ad3b-3ecbc00357c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:13.689686 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.689485    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-cabundle-cert" pod "91d86fc4-170a-43d7-ad3b-3ecbc00357c0" (UID: "91d86fc4-170a-43d7-ad3b-3ecbc00357c0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:13.689686 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.689590    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kube-rbac-proxy-sar-config" pod "91d86fc4-170a-43d7-ad3b-3ecbc00357c0" (UID: "91d86fc4-170a-43d7-ad3b-3ecbc00357c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:13.691242 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.691216    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-proxy-tls" pod "91d86fc4-170a-43d7-ad3b-3ecbc00357c0" (UID: "91d86fc4-170a-43d7-ad3b-3ecbc00357c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:13.691360 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.691342    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kube-api-access-g8jk5" pod "91d86fc4-170a-43d7-ad3b-3ecbc00357c0" (UID: "91d86fc4-170a-43d7-ad3b-3ecbc00357c0"). InnerVolumeSpecName "kube-api-access-g8jk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:13.789957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.789929    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:13.789957 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.789956    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:13.790163 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.789966    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:13.790163 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.789975    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8jk5\" (UniqueName: \"kubernetes.io/projected/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-kube-api-access-g8jk5\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:13.790163 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:13.789985    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91d86fc4-170a-43d7-ad3b-3ecbc00357c0-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:14.011564 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:14.011482    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"]
Oct 30 19:34:14.017581 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:14.017550    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-v2-runtime-predictor-744b6d5b84-wxlx2"]
Oct 30 19:34:15.656955 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:15.656922    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d86fc4-170a-43d7-ad3b-3ecbc00357c0" path="/var/lib/kubelet/pods/91d86fc4-170a-43d7-ad3b-3ecbc00357c0/volumes"
Oct 30 19:34:17.617171 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.617117    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"]
Oct 30 19:34:17.617543 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.617411    4511 cpu_manager.go:414] "containerMap: removing container" podUID="91d86fc4-170a-43d7-ad3b-3ecbc00357c0" containerName="storage-initializer"
Oct 30 19:34:17.617543 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.617421    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d86fc4-170a-43d7-ad3b-3ecbc00357c0" containerName="storage-initializer"
Oct 30 19:34:17.617543 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.617443    4511 cpu_manager.go:414] "containerMap: removing container" podUID="91d86fc4-170a-43d7-ad3b-3ecbc00357c0" containerName="kserve-container"
Oct 30 19:34:17.617543 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.617447    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d86fc4-170a-43d7-ad3b-3ecbc00357c0" containerName="kserve-container"
Oct 30 19:34:17.617543 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.617456    4511 cpu_manager.go:414] "containerMap: removing container" podUID="91d86fc4-170a-43d7-ad3b-3ecbc00357c0" containerName="kube-rbac-proxy"
Oct 30 19:34:17.617543 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.617461    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d86fc4-170a-43d7-ad3b-3ecbc00357c0" containerName="kube-rbac-proxy"
Oct 30 19:34:17.617543 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.617504    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d86fc4-170a-43d7-ad3b-3ecbc00357c0" containerName="kube-rbac-proxy"
Oct 30 19:34:17.617543 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.617516    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d86fc4-170a-43d7-ad3b-3ecbc00357c0" containerName="kserve-container"
Oct 30 19:34:17.622346 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.622326    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:17.672724 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.672699    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:34:17.716893 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.716861    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mlwk\" (UniqueName: \"kubernetes.io/projected/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kube-api-access-7mlwk\") pod \"isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:17.717029 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.716918    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:17.717029 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.716954    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:17.717029 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.716988    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:17.717216 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.717050    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:17.720247 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.720223    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Oct 30 19:34:17.720371 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.720324    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:34:17.720553 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.720534    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:34:17.720654 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.720642    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:34:17.720721 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.720695    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:34:17.720848 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.720833    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Oct 30 19:34:17.724691 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.724668    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"]
Oct 30 19:34:17.818233 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.818203    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:17.818233 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.818235    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:17.818460 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.818269    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:17.818460 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.818333    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mlwk\" (UniqueName: \"kubernetes.io/projected/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kube-api-access-7mlwk\") pod \"isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:17.818460 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.818360    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:17.818721 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.818699    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:17.819085 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.819062    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:17.819157 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.819062    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:17.821092 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.821074    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:17.877377 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.877331    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mlwk\" (UniqueName: \"kubernetes.io/projected/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kube-api-access-7mlwk\") pod \"isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") " pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:17.939413 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:17.939391    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:18.359953 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:18.359922    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"]
Oct 30 19:34:18.367065 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:34:18.367038    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ffb2db_80cc_4fb1_95cc_a4e6aa367474.slice/crio-5e06ccc9b7cae7199f7e6007dafab93206363ef4efae8f13680f6904cd059723 WatchSource:0}: Error finding container 5e06ccc9b7cae7199f7e6007dafab93206363ef4efae8f13680f6904cd059723: Status 404 returned error can't find the container with id 5e06ccc9b7cae7199f7e6007dafab93206363ef4efae8f13680f6904cd059723
Oct 30 19:34:18.617256 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:18.617162    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm" event={"ID":"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474","Type":"ContainerStarted","Data":"a28cc70e8d04524360eb1411735394d5904a00ac6ac89e097d5e6c62c35ce8b8"}
Oct 30 19:34:18.617256 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:18.617222    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm" event={"ID":"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474","Type":"ContainerStarted","Data":"5e06ccc9b7cae7199f7e6007dafab93206363ef4efae8f13680f6904cd059723"}
Oct 30 19:34:22.633954 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:22.633924    4511 generic.go:356] "Generic (PLEG): container finished" podID="a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" containerID="a28cc70e8d04524360eb1411735394d5904a00ac6ac89e097d5e6c62c35ce8b8" exitCode=0
Oct 30 19:34:22.634348 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:22.633999    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm" event={"ID":"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474","Type":"ContainerDied","Data":"a28cc70e8d04524360eb1411735394d5904a00ac6ac89e097d5e6c62c35ce8b8"}
Oct 30 19:34:23.640062 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:23.640027    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm" event={"ID":"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474","Type":"ContainerStarted","Data":"b4a363498ad9cb7a39ade736f3f690bb43090d83538b63fec494c2ace29aa6a5"}
Oct 30 19:34:23.640596 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:23.640114    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm" event={"ID":"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474","Type":"ContainerStarted","Data":"c5b4cef26d6146b011881d6f93ee6d0422f933f6615167cf33aa5747e374a13a"}
Oct 30 19:34:23.640596 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:23.640404    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:23.640596 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:23.640533    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:23.641806 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:23.641777    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm" podUID="a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Oct 30 19:34:23.756322 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:23.756277    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm" podStartSLOduration=2.937049602 podStartE2EDuration="6.756265742s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.81921614s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:23.755847495 +0000 UTC m=+3358.273532407" watchObservedRunningTime="2026-10-30 19:34:23.756265742 +0000 UTC m=+3358.273950642"
Oct 30 19:34:24.644185 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:24.644150    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm" podUID="a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Oct 30 19:34:29.649685 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:29.649645    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:29.650288 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:29.650260    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm" podUID="a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Oct 30 19:34:39.655609 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:39.655541    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:40.156609 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:34:40.156570    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:34:47.472047 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.472014    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"]
Oct 30 19:34:47.475411 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.475392    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:34:47.525831 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.525807    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:34:47.538179 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.538158    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71ba8320-c44f-41fe-8420-f59be98dc154-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-6f4576b95c-fcbnm\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") " pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:34:47.538242 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.538188    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71ba8320-c44f-41fe-8420-f59be98dc154-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-6f4576b95c-fcbnm\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") " pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:34:47.538242 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.538211    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71ba8320-c44f-41fe-8420-f59be98dc154-proxy-tls\") pod \"isvc-paddle-runtime-predictor-6f4576b95c-fcbnm\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") " pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:34:47.538242 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.538228    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71ba8320-c44f-41fe-8420-f59be98dc154-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-6f4576b95c-fcbnm\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") " pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:34:47.538352 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.538333    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt6rf\" (UniqueName: \"kubernetes.io/projected/71ba8320-c44f-41fe-8420-f59be98dc154-kube-api-access-gt6rf\") pod \"isvc-paddle-runtime-predictor-6f4576b95c-fcbnm\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") " pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:34:47.573081 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.573058    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:34:47.573199 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.573089    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:34:47.573275 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.573259    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:34:47.573329 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.573274    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Oct 30 19:34:47.573329 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.573314    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:34:47.573401 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.573336    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:34:47.574639 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.574621    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"]
Oct 30 19:34:47.639201 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.639172    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71ba8320-c44f-41fe-8420-f59be98dc154-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-6f4576b95c-fcbnm\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") " pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:34:47.639201 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.639203    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71ba8320-c44f-41fe-8420-f59be98dc154-proxy-tls\") pod \"isvc-paddle-runtime-predictor-6f4576b95c-fcbnm\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") " pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:34:47.639369 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.639224    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71ba8320-c44f-41fe-8420-f59be98dc154-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-6f4576b95c-fcbnm\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") " pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:34:47.639369 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.639256    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt6rf\" (UniqueName: \"kubernetes.io/projected/71ba8320-c44f-41fe-8420-f59be98dc154-kube-api-access-gt6rf\") pod \"isvc-paddle-runtime-predictor-6f4576b95c-fcbnm\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") " pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:34:47.639369 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.639331    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71ba8320-c44f-41fe-8420-f59be98dc154-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-6f4576b95c-fcbnm\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") " pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:34:47.639706 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.639685    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71ba8320-c44f-41fe-8420-f59be98dc154-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-6f4576b95c-fcbnm\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") " pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:34:47.640026 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.639995    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71ba8320-c44f-41fe-8420-f59be98dc154-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-6f4576b95c-fcbnm\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") " pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:34:47.640159 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.639999    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71ba8320-c44f-41fe-8420-f59be98dc154-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-6f4576b95c-fcbnm\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") " pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:34:47.642067 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.642046    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71ba8320-c44f-41fe-8420-f59be98dc154-proxy-tls\") pod \"isvc-paddle-runtime-predictor-6f4576b95c-fcbnm\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") " pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:34:47.698757 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.698735    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt6rf\" (UniqueName: \"kubernetes.io/projected/71ba8320-c44f-41fe-8420-f59be98dc154-kube-api-access-gt6rf\") pod \"isvc-paddle-runtime-predictor-6f4576b95c-fcbnm\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") " pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:34:47.792322 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:47.792259    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:34:48.212611 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:48.212582    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"]
Oct 30 19:34:48.219185 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:34:48.219150    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ba8320_c44f_41fe_8420_f59be98dc154.slice/crio-cb0525802d6246ef951da1e6f729a943e13bb57170fdeb8d178f96869fa996b6 WatchSource:0}: Error finding container cb0525802d6246ef951da1e6f729a943e13bb57170fdeb8d178f96869fa996b6: Status 404 returned error can't find the container with id cb0525802d6246ef951da1e6f729a943e13bb57170fdeb8d178f96869fa996b6
Oct 30 19:34:48.495625 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:48.495551    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"]
Oct 30 19:34:48.496010 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:48.495865    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm" podUID="a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" containerName="kserve-container" containerID="cri-o://c5b4cef26d6146b011881d6f93ee6d0422f933f6615167cf33aa5747e374a13a" gracePeriod=30
Oct 30 19:34:48.496010 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:48.495947    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm" podUID="a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" containerName="kube-rbac-proxy" containerID="cri-o://b4a363498ad9cb7a39ade736f3f690bb43090d83538b63fec494c2ace29aa6a5" gracePeriod=30
Oct 30 19:34:48.734387 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:48.734349    4511 generic.go:356] "Generic (PLEG): container finished" podID="a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" containerID="b4a363498ad9cb7a39ade736f3f690bb43090d83538b63fec494c2ace29aa6a5" exitCode=0
Oct 30 19:34:48.734568 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:48.734384    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm" event={"ID":"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474","Type":"ContainerDied","Data":"b4a363498ad9cb7a39ade736f3f690bb43090d83538b63fec494c2ace29aa6a5"}
Oct 30 19:34:48.736239 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:48.736215    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" event={"ID":"71ba8320-c44f-41fe-8420-f59be98dc154","Type":"ContainerStarted","Data":"519b5414e78bdec6b8a815f001e04ffbf1250aa09ec27d8765f42ea8ba6e2b45"}
Oct 30 19:34:48.736384 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:48.736261    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" event={"ID":"71ba8320-c44f-41fe-8420-f59be98dc154","Type":"ContainerStarted","Data":"cb0525802d6246ef951da1e6f729a943e13bb57170fdeb8d178f96869fa996b6"}
Oct 30 19:34:49.235946 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.235922    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:49.251972 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.251949    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kube-rbac-proxy-sar-config\") pod \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") "
Oct 30 19:34:49.252090 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.252007    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-cabundle-cert\") pod \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") "
Oct 30 19:34:49.252090 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.252039    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kube-api-access-7mlwk\" (UniqueName: \"kubernetes.io/projected/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kube-api-access-7mlwk\") pod \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") "
Oct 30 19:34:49.252090 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.252057    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kserve-provision-location\") pod \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") "
Oct 30 19:34:49.252337 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.252301    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kube-rbac-proxy-sar-config" pod "a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" (UID: "a8ffb2db-80cc-4fb1-95cc-a4e6aa367474"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:49.252447 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.252424    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-cabundle-cert" pod "a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" (UID: "a8ffb2db-80cc-4fb1-95cc-a4e6aa367474"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:49.252500 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.252463    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kserve-provision-location" pod "a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" (UID: "a8ffb2db-80cc-4fb1-95cc-a4e6aa367474"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:49.254171 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.254152    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kube-api-access-7mlwk" pod "a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" (UID: "a8ffb2db-80cc-4fb1-95cc-a4e6aa367474"). InnerVolumeSpecName "kube-api-access-7mlwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:49.352851 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.352784    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-proxy-tls\") pod \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\" (UID: \"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474\") "
Oct 30 19:34:49.352978 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.352962    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mlwk\" (UniqueName: \"kubernetes.io/projected/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kube-api-access-7mlwk\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:49.353030 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.352985    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:49.353030 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.353000    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:49.353030 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.353015    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:49.354878 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.354857    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-proxy-tls" pod "a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" (UID: "a8ffb2db-80cc-4fb1-95cc-a4e6aa367474"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:49.453433 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.453406    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:49.741799 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.741767    4511 generic.go:356] "Generic (PLEG): container finished" podID="a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" containerID="c5b4cef26d6146b011881d6f93ee6d0422f933f6615167cf33aa5747e374a13a" exitCode=0
Oct 30 19:34:49.742284 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.741824    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"
Oct 30 19:34:49.742284 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.741863    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm" event={"ID":"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474","Type":"ContainerDied","Data":"c5b4cef26d6146b011881d6f93ee6d0422f933f6615167cf33aa5747e374a13a"}
Oct 30 19:34:49.742284 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.741939    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm" event={"ID":"a8ffb2db-80cc-4fb1-95cc-a4e6aa367474","Type":"ContainerDied","Data":"5e06ccc9b7cae7199f7e6007dafab93206363ef4efae8f13680f6904cd059723"}
Oct 30 19:34:49.742284 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.741981    4511 scope.go:122] "RemoveContainer" containerID="b4a363498ad9cb7a39ade736f3f690bb43090d83538b63fec494c2ace29aa6a5"
Oct 30 19:34:49.750189 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.750173    4511 scope.go:122] "RemoveContainer" containerID="c5b4cef26d6146b011881d6f93ee6d0422f933f6615167cf33aa5747e374a13a"
Oct 30 19:34:49.757340 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.757323    4511 scope.go:122] "RemoveContainer" containerID="a28cc70e8d04524360eb1411735394d5904a00ac6ac89e097d5e6c62c35ce8b8"
Oct 30 19:34:49.764187 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.764171    4511 scope.go:122] "RemoveContainer" containerID="b4a363498ad9cb7a39ade736f3f690bb43090d83538b63fec494c2ace29aa6a5"
Oct 30 19:34:49.764440 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:34:49.764422    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a363498ad9cb7a39ade736f3f690bb43090d83538b63fec494c2ace29aa6a5\": container with ID starting with b4a363498ad9cb7a39ade736f3f690bb43090d83538b63fec494c2ace29aa6a5 not found: ID does not exist" containerID="b4a363498ad9cb7a39ade736f3f690bb43090d83538b63fec494c2ace29aa6a5"
Oct 30 19:34:49.764514 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.764452    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a363498ad9cb7a39ade736f3f690bb43090d83538b63fec494c2ace29aa6a5"} err="failed to get container status \"b4a363498ad9cb7a39ade736f3f690bb43090d83538b63fec494c2ace29aa6a5\": rpc error: code = NotFound desc = could not find container \"b4a363498ad9cb7a39ade736f3f690bb43090d83538b63fec494c2ace29aa6a5\": container with ID starting with b4a363498ad9cb7a39ade736f3f690bb43090d83538b63fec494c2ace29aa6a5 not found: ID does not exist"
Oct 30 19:34:49.764514 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.764476    4511 scope.go:122] "RemoveContainer" containerID="c5b4cef26d6146b011881d6f93ee6d0422f933f6615167cf33aa5747e374a13a"
Oct 30 19:34:49.764737 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:34:49.764719    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b4cef26d6146b011881d6f93ee6d0422f933f6615167cf33aa5747e374a13a\": container with ID starting with c5b4cef26d6146b011881d6f93ee6d0422f933f6615167cf33aa5747e374a13a not found: ID does not exist" containerID="c5b4cef26d6146b011881d6f93ee6d0422f933f6615167cf33aa5747e374a13a"
Oct 30 19:34:49.764788 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.764743    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b4cef26d6146b011881d6f93ee6d0422f933f6615167cf33aa5747e374a13a"} err="failed to get container status \"c5b4cef26d6146b011881d6f93ee6d0422f933f6615167cf33aa5747e374a13a\": rpc error: code = NotFound desc = could not find container \"c5b4cef26d6146b011881d6f93ee6d0422f933f6615167cf33aa5747e374a13a\": container with ID starting with c5b4cef26d6146b011881d6f93ee6d0422f933f6615167cf33aa5747e374a13a not found: ID does not exist"
Oct 30 19:34:49.764788 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.764763    4511 scope.go:122] "RemoveContainer" containerID="a28cc70e8d04524360eb1411735394d5904a00ac6ac89e097d5e6c62c35ce8b8"
Oct 30 19:34:49.764992 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:34:49.764975    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a28cc70e8d04524360eb1411735394d5904a00ac6ac89e097d5e6c62c35ce8b8\": container with ID starting with a28cc70e8d04524360eb1411735394d5904a00ac6ac89e097d5e6c62c35ce8b8 not found: ID does not exist" containerID="a28cc70e8d04524360eb1411735394d5904a00ac6ac89e097d5e6c62c35ce8b8"
Oct 30 19:34:49.765044 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.765001    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a28cc70e8d04524360eb1411735394d5904a00ac6ac89e097d5e6c62c35ce8b8"} err="failed to get container status \"a28cc70e8d04524360eb1411735394d5904a00ac6ac89e097d5e6c62c35ce8b8\": rpc error: code = NotFound desc = could not find container \"a28cc70e8d04524360eb1411735394d5904a00ac6ac89e097d5e6c62c35ce8b8\": container with ID starting with a28cc70e8d04524360eb1411735394d5904a00ac6ac89e097d5e6c62c35ce8b8 not found: ID does not exist"
Oct 30 19:34:49.900308 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.900248    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"]
Oct 30 19:34:49.947498 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:49.947473    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-lightgbm-v2-kserve-predictor-6bbd4567f9-56htm"]
Oct 30 19:34:51.656023 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:51.655989    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" path="/var/lib/kubelet/pods/a8ffb2db-80cc-4fb1-95cc-a4e6aa367474/volumes"
Oct 30 19:34:52.748122 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.748085    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"]
Oct 30 19:34:52.748516 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.748420    4511 cpu_manager.go:414] "containerMap: removing container" podUID="a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" containerName="kserve-container"
Oct 30 19:34:52.748516 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.748433    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" containerName="kserve-container"
Oct 30 19:34:52.748516 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.748457    4511 cpu_manager.go:414] "containerMap: removing container" podUID="a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" containerName="storage-initializer"
Oct 30 19:34:52.748516 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.748463    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" containerName="storage-initializer"
Oct 30 19:34:52.748516 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.748469    4511 cpu_manager.go:414] "containerMap: removing container" podUID="a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" containerName="kube-rbac-proxy"
Oct 30 19:34:52.748516 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.748474    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" containerName="kube-rbac-proxy"
Oct 30 19:34:52.748702 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.748522    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" containerName="kube-rbac-proxy"
Oct 30 19:34:52.748702 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.748533    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ffb2db-80cc-4fb1-95cc-a4e6aa367474" containerName="kserve-container"
Oct 30 19:34:52.753538 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.753516    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:52.775719 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.775691    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/718096c0-6223-4d5c-87ac-95bb6a460bb9-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") " pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:52.775841 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.775740    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/718096c0-6223-4d5c-87ac-95bb6a460bb9-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") " pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:52.775841 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.775760    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/718096c0-6223-4d5c-87ac-95bb6a460bb9-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") " pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:52.775975 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.775914    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx49q\" (UniqueName: \"kubernetes.io/projected/718096c0-6223-4d5c-87ac-95bb6a460bb9-kube-api-access-gx49q\") pod \"isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") " pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:52.775975 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.775960    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/718096c0-6223-4d5c-87ac-95bb6a460bb9-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") " pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:52.851298 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.851275    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:34:52.851440 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.851312    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Oct 30 19:34:52.851440 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.851332    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:34:52.851440 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.851353    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:34:52.851440 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.851389    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:34:52.851440 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.851392    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:34:52.851656 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.851465    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:34:52.853423 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.853404    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"]
Oct 30 19:34:52.876803 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.876781    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/718096c0-6223-4d5c-87ac-95bb6a460bb9-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") " pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:52.876924 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.876822    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/718096c0-6223-4d5c-87ac-95bb6a460bb9-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") " pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:52.876924 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.876842    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/718096c0-6223-4d5c-87ac-95bb6a460bb9-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") " pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:52.876924 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.876885    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx49q\" (UniqueName: \"kubernetes.io/projected/718096c0-6223-4d5c-87ac-95bb6a460bb9-kube-api-access-gx49q\") pod \"isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") " pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:52.876924 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.876908    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/718096c0-6223-4d5c-87ac-95bb6a460bb9-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") " pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:52.877322 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.877286    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/718096c0-6223-4d5c-87ac-95bb6a460bb9-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") " pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:52.877567 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.877548    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/718096c0-6223-4d5c-87ac-95bb6a460bb9-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") " pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:52.877656 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.877592    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/718096c0-6223-4d5c-87ac-95bb6a460bb9-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") " pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:52.879486 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.879467    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/718096c0-6223-4d5c-87ac-95bb6a460bb9-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") " pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:52.936543 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:52.936510    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx49q\" (UniqueName: \"kubernetes.io/projected/718096c0-6223-4d5c-87ac-95bb6a460bb9-kube-api-access-gx49q\") pod \"isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") " pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:53.072672 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:53.072645    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:53.494028 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:53.494002    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"]
Oct 30 19:34:53.516938 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:34:53.516914    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod718096c0_6223_4d5c_87ac_95bb6a460bb9.slice/crio-5841b42b7b2be2a5afe50ed8de1e52818f3d75121b6fbb43e36c8f5668d1404a WatchSource:0}: Error finding container 5841b42b7b2be2a5afe50ed8de1e52818f3d75121b6fbb43e36c8f5668d1404a: Status 404 returned error can't find the container with id 5841b42b7b2be2a5afe50ed8de1e52818f3d75121b6fbb43e36c8f5668d1404a
Oct 30 19:34:53.760080 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:53.760011    4511 generic.go:356] "Generic (PLEG): container finished" podID="71ba8320-c44f-41fe-8420-f59be98dc154" containerID="519b5414e78bdec6b8a815f001e04ffbf1250aa09ec27d8765f42ea8ba6e2b45" exitCode=0
Oct 30 19:34:53.760530 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:53.760081    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" event={"ID":"71ba8320-c44f-41fe-8420-f59be98dc154","Type":"ContainerDied","Data":"519b5414e78bdec6b8a815f001e04ffbf1250aa09ec27d8765f42ea8ba6e2b45"}
Oct 30 19:34:53.761608 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:53.761591    4511 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:34:53.761799 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:53.761774    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm" event={"ID":"718096c0-6223-4d5c-87ac-95bb6a460bb9","Type":"ContainerStarted","Data":"4eb68086d667631780393352f25ec8ed54e74b1a924fc944bb7f54714c8d93ed"}
Oct 30 19:34:53.761860 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:53.761823    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm" event={"ID":"718096c0-6223-4d5c-87ac-95bb6a460bb9","Type":"ContainerStarted","Data":"5841b42b7b2be2a5afe50ed8de1e52818f3d75121b6fbb43e36c8f5668d1404a"}
Oct 30 19:34:57.781378 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:57.781341    4511 generic.go:356] "Generic (PLEG): container finished" podID="718096c0-6223-4d5c-87ac-95bb6a460bb9" containerID="4eb68086d667631780393352f25ec8ed54e74b1a924fc944bb7f54714c8d93ed" exitCode=0
Oct 30 19:34:57.781758 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:57.781403    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm" event={"ID":"718096c0-6223-4d5c-87ac-95bb6a460bb9","Type":"ContainerDied","Data":"4eb68086d667631780393352f25ec8ed54e74b1a924fc944bb7f54714c8d93ed"}
Oct 30 19:34:58.789300 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:58.789258    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm" event={"ID":"718096c0-6223-4d5c-87ac-95bb6a460bb9","Type":"ContainerStarted","Data":"547652a3c91c45956f88fd563ae262395f388be60869ff4df302849d5b2161c3"}
Oct 30 19:34:58.789899 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:58.789339    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm" event={"ID":"718096c0-6223-4d5c-87ac-95bb6a460bb9","Type":"ContainerStarted","Data":"9156ab54ac6655c2621670fb7b6aa6039d4f6bcc9b903c26cc3deaa2bda41eae"}
Oct 30 19:34:58.789899 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:58.789567    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:34:58.939871 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:58.939806    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm" podStartSLOduration=2.882024785 podStartE2EDuration="6.939786729s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.057761944s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:58.938514324 +0000 UTC m=+3393.456199231" watchObservedRunningTime="2026-10-30 19:34:58.939786729 +0000 UTC m=+3393.457471630"
Oct 30 19:34:59.794649 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:34:59.794332    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:35:05.805769 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:05.805734    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:35:07.829711 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:07.829680    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" event={"ID":"71ba8320-c44f-41fe-8420-f59be98dc154","Type":"ContainerStarted","Data":"be98d418410fc0006e721fac7ec3a0402774438e3eb4e0ad1ef3db456e4aed76"}
Oct 30 19:35:07.830095 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:07.829739    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" event={"ID":"71ba8320-c44f-41fe-8420-f59be98dc154","Type":"ContainerStarted","Data":"895d78acde0b504225229ba79fae98d46bbaffde7c58dd69b085195c5b91a29c"}
Oct 30 19:35:07.830095 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:07.829931    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:35:07.995040 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:07.994992    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" podStartSLOduration=2.78172328 podStartE2EDuration="20.994979581s" totalImagesPullingTime="13.03783071s" totalInitContainerRuntime="5.175425591s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:07.993305782 +0000 UTC m=+3402.510990682" watchObservedRunningTime="2026-10-30 19:35:07.994979581 +0000 UTC m=+3402.512664481"
Oct 30 19:35:08.833830 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:08.833797    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:35:08.835018 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:08.834988    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Oct 30 19:35:09.838135 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:09.838098    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Oct 30 19:35:10.156793 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:35:10.156706    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:35:14.842954 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:14.842926    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:35:14.843463 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:14.843436    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Oct 30 19:35:24.843699 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:24.843660    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Oct 30 19:35:34.843704 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:34.843668    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Oct 30 19:35:35.810480 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:35.810447    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:35:40.156805 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:35:40.156773    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:35:42.880591 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:42.880560    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"]
Oct 30 19:35:42.881003 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:42.880774    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm" podUID="718096c0-6223-4d5c-87ac-95bb6a460bb9" containerName="kserve-container" containerID="cri-o://9156ab54ac6655c2621670fb7b6aa6039d4f6bcc9b903c26cc3deaa2bda41eae" gracePeriod=30
Oct 30 19:35:42.881003 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:42.880787    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm" podUID="718096c0-6223-4d5c-87ac-95bb6a460bb9" containerName="kube-rbac-proxy" containerID="cri-o://547652a3c91c45956f88fd563ae262395f388be60869ff4df302849d5b2161c3" gracePeriod=30
Oct 30 19:35:43.972183 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:43.972153    4511 generic.go:356] "Generic (PLEG): container finished" podID="718096c0-6223-4d5c-87ac-95bb6a460bb9" containerID="547652a3c91c45956f88fd563ae262395f388be60869ff4df302849d5b2161c3" exitCode=0
Oct 30 19:35:43.972183 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:43.972178    4511 generic.go:356] "Generic (PLEG): container finished" podID="718096c0-6223-4d5c-87ac-95bb6a460bb9" containerID="9156ab54ac6655c2621670fb7b6aa6039d4f6bcc9b903c26cc3deaa2bda41eae" exitCode=0
Oct 30 19:35:43.972577 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:43.972230    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm" event={"ID":"718096c0-6223-4d5c-87ac-95bb6a460bb9","Type":"ContainerDied","Data":"547652a3c91c45956f88fd563ae262395f388be60869ff4df302849d5b2161c3"}
Oct 30 19:35:43.972577 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:43.972304    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm" event={"ID":"718096c0-6223-4d5c-87ac-95bb6a460bb9","Type":"ContainerDied","Data":"9156ab54ac6655c2621670fb7b6aa6039d4f6bcc9b903c26cc3deaa2bda41eae"}
Oct 30 19:35:44.024913 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.024892    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:35:44.187754 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.187662    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/718096c0-6223-4d5c-87ac-95bb6a460bb9-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/718096c0-6223-4d5c-87ac-95bb6a460bb9-kserve-provision-location\") pod \"718096c0-6223-4d5c-87ac-95bb6a460bb9\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") "
Oct 30 19:35:44.187754 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.187699    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/718096c0-6223-4d5c-87ac-95bb6a460bb9-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/718096c0-6223-4d5c-87ac-95bb6a460bb9-cabundle-cert\") pod \"718096c0-6223-4d5c-87ac-95bb6a460bb9\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") "
Oct 30 19:35:44.187996 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.187762    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/718096c0-6223-4d5c-87ac-95bb6a460bb9-kube-api-access-gx49q\" (UniqueName: \"kubernetes.io/projected/718096c0-6223-4d5c-87ac-95bb6a460bb9-kube-api-access-gx49q\") pod \"718096c0-6223-4d5c-87ac-95bb6a460bb9\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") "
Oct 30 19:35:44.187996 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.187796    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/718096c0-6223-4d5c-87ac-95bb6a460bb9-proxy-tls\" (UniqueName: \"kubernetes.io/secret/718096c0-6223-4d5c-87ac-95bb6a460bb9-proxy-tls\") pod \"718096c0-6223-4d5c-87ac-95bb6a460bb9\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") "
Oct 30 19:35:44.187996 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.187829    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/718096c0-6223-4d5c-87ac-95bb6a460bb9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/718096c0-6223-4d5c-87ac-95bb6a460bb9-kube-rbac-proxy-sar-config\") pod \"718096c0-6223-4d5c-87ac-95bb6a460bb9\" (UID: \"718096c0-6223-4d5c-87ac-95bb6a460bb9\") "
Oct 30 19:35:44.188180 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.188069    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/718096c0-6223-4d5c-87ac-95bb6a460bb9-kserve-provision-location" pod "718096c0-6223-4d5c-87ac-95bb6a460bb9" (UID: "718096c0-6223-4d5c-87ac-95bb6a460bb9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:44.188281 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.188255    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718096c0-6223-4d5c-87ac-95bb6a460bb9-cabundle-cert" pod "718096c0-6223-4d5c-87ac-95bb6a460bb9" (UID: "718096c0-6223-4d5c-87ac-95bb6a460bb9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:44.188281 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.188262    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718096c0-6223-4d5c-87ac-95bb6a460bb9-kube-rbac-proxy-sar-config" pod "718096c0-6223-4d5c-87ac-95bb6a460bb9" (UID: "718096c0-6223-4d5c-87ac-95bb6a460bb9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:44.190307 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.190276    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/718096c0-6223-4d5c-87ac-95bb6a460bb9-kube-api-access-gx49q" pod "718096c0-6223-4d5c-87ac-95bb6a460bb9" (UID: "718096c0-6223-4d5c-87ac-95bb6a460bb9"). InnerVolumeSpecName "kube-api-access-gx49q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:44.190396 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.190324    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/718096c0-6223-4d5c-87ac-95bb6a460bb9-proxy-tls" pod "718096c0-6223-4d5c-87ac-95bb6a460bb9" (UID: "718096c0-6223-4d5c-87ac-95bb6a460bb9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:44.288398 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.288350    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/718096c0-6223-4d5c-87ac-95bb6a460bb9-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:44.288398 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.288393    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/718096c0-6223-4d5c-87ac-95bb6a460bb9-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:44.288398 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.288403    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx49q\" (UniqueName: \"kubernetes.io/projected/718096c0-6223-4d5c-87ac-95bb6a460bb9-kube-api-access-gx49q\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:44.288398 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.288413    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/718096c0-6223-4d5c-87ac-95bb6a460bb9-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:44.288666 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.288423    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/718096c0-6223-4d5c-87ac-95bb6a460bb9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:44.843789 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.843754    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Oct 30 19:35:44.978070 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.978044    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"
Oct 30 19:35:44.978070 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.978054    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm" event={"ID":"718096c0-6223-4d5c-87ac-95bb6a460bb9","Type":"ContainerDied","Data":"5841b42b7b2be2a5afe50ed8de1e52818f3d75121b6fbb43e36c8f5668d1404a"}
Oct 30 19:35:44.978554 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.978115    4511 scope.go:122] "RemoveContainer" containerID="547652a3c91c45956f88fd563ae262395f388be60869ff4df302849d5b2161c3"
Oct 30 19:35:44.987073 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.987051    4511 scope.go:122] "RemoveContainer" containerID="9156ab54ac6655c2621670fb7b6aa6039d4f6bcc9b903c26cc3deaa2bda41eae"
Oct 30 19:35:44.994663 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:44.994646    4511 scope.go:122] "RemoveContainer" containerID="4eb68086d667631780393352f25ec8ed54e74b1a924fc944bb7f54714c8d93ed"
Oct 30 19:35:45.184499 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:45.184420    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"]
Oct 30 19:35:45.231970 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:45.231941    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-mlflow-v2-runtime-predictor-5846c4c5d5-l64mm"]
Oct 30 19:35:45.656243 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:45.656215    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="718096c0-6223-4d5c-87ac-95bb6a460bb9" path="/var/lib/kubelet/pods/718096c0-6223-4d5c-87ac-95bb6a460bb9/volumes"
Oct 30 19:35:47.899531 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:47.899496    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"]
Oct 30 19:35:47.899897 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:47.899806    4511 cpu_manager.go:414] "containerMap: removing container" podUID="718096c0-6223-4d5c-87ac-95bb6a460bb9" containerName="storage-initializer"
Oct 30 19:35:47.899897 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:47.899819    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="718096c0-6223-4d5c-87ac-95bb6a460bb9" containerName="storage-initializer"
Oct 30 19:35:47.899897 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:47.899838    4511 cpu_manager.go:414] "containerMap: removing container" podUID="718096c0-6223-4d5c-87ac-95bb6a460bb9" containerName="kserve-container"
Oct 30 19:35:47.899897 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:47.899844    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="718096c0-6223-4d5c-87ac-95bb6a460bb9" containerName="kserve-container"
Oct 30 19:35:47.899897 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:47.899862    4511 cpu_manager.go:414] "containerMap: removing container" podUID="718096c0-6223-4d5c-87ac-95bb6a460bb9" containerName="kube-rbac-proxy"
Oct 30 19:35:47.899897 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:47.899868    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="718096c0-6223-4d5c-87ac-95bb6a460bb9" containerName="kube-rbac-proxy"
Oct 30 19:35:47.900079 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:47.899921    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="718096c0-6223-4d5c-87ac-95bb6a460bb9" containerName="kserve-container"
Oct 30 19:35:47.900079 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:47.899933    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="718096c0-6223-4d5c-87ac-95bb6a460bb9" containerName="kube-rbac-proxy"
Oct 30 19:35:47.903066 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:47.903049    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:47.914733 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:47.914708    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-56b5dcd664-wlww6\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:47.914828 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:47.914747    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-56b5dcd664-wlww6\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:47.914892 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:47.914867    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-56b5dcd664-wlww6\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:47.914932 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:47.914919    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-56b5dcd664-wlww6\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:47.915031 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:47.915017    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqrhx\" (UniqueName: \"kubernetes.io/projected/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kube-api-access-lqrhx\") pod \"isvc-sklearn-mcp-predictor-56b5dcd664-wlww6\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:47.954246 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:47.954225    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:35:47.961098 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:47.961071    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:35:48.001459 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.001442    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:35:48.001566 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.001465    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:35:48.001566 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.001496    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Oct 30 19:35:48.001566 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.001545    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Oct 30 19:35:48.001566 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.001554    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:35:48.009682 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.009661    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"]
Oct 30 19:35:48.016421 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.016396    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-56b5dcd664-wlww6\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:48.016518 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.016446    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-56b5dcd664-wlww6\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:48.016518 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.016503    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqrhx\" (UniqueName: \"kubernetes.io/projected/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kube-api-access-lqrhx\") pod \"isvc-sklearn-mcp-predictor-56b5dcd664-wlww6\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:48.016652 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.016631    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-56b5dcd664-wlww6\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:48.016725 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.016678    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-56b5dcd664-wlww6\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:48.016833 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.016807    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-56b5dcd664-wlww6\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:48.017205 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.017184    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-56b5dcd664-wlww6\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:48.017349 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.017331    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-56b5dcd664-wlww6\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:48.019245 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.019225    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-56b5dcd664-wlww6\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:48.075991 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.075966    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqrhx\" (UniqueName: \"kubernetes.io/projected/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kube-api-access-lqrhx\") pod \"isvc-sklearn-mcp-predictor-56b5dcd664-wlww6\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") " pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:48.217531 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.217465    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:48.641486 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.641454    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"]
Oct 30 19:35:48.649177 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:35:48.649146    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd4222c_cd81_4480_8129_a6f5cd31ab4d.slice/crio-d471609811ea7a883be224ef8c9886d513cbbd4a9a21a68331b9ada0c30b9913 WatchSource:0}: Error finding container d471609811ea7a883be224ef8c9886d513cbbd4a9a21a68331b9ada0c30b9913: Status 404 returned error can't find the container with id d471609811ea7a883be224ef8c9886d513cbbd4a9a21a68331b9ada0c30b9913
Oct 30 19:35:48.994988 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.994902    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" event={"ID":"6dd4222c-cd81-4480-8129-a6f5cd31ab4d","Type":"ContainerStarted","Data":"64442bb77921246e6a4e1b77d1dd1d46d45de4c382ee77c164fd5312a5965632"}
Oct 30 19:35:48.995363 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:48.994987    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" event={"ID":"6dd4222c-cd81-4480-8129-a6f5cd31ab4d","Type":"ContainerStarted","Data":"d471609811ea7a883be224ef8c9886d513cbbd4a9a21a68331b9ada0c30b9913"}
Oct 30 19:35:53.011953 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:53.011921    4511 generic.go:356] "Generic (PLEG): container finished" podID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerID="64442bb77921246e6a4e1b77d1dd1d46d45de4c382ee77c164fd5312a5965632" exitCode=0
Oct 30 19:35:53.012401 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:53.011981    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" event={"ID":"6dd4222c-cd81-4480-8129-a6f5cd31ab4d","Type":"ContainerDied","Data":"64442bb77921246e6a4e1b77d1dd1d46d45de4c382ee77c164fd5312a5965632"}
Oct 30 19:35:54.018733 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:54.018691    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" event={"ID":"6dd4222c-cd81-4480-8129-a6f5cd31ab4d","Type":"ContainerStarted","Data":"7869af4924ee0d41d2ebc9930eab236389d958602b8b987d2da03317878331fe"}
Oct 30 19:35:54.844280 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:54.844251    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:35:59.043711 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:59.043671    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" event={"ID":"6dd4222c-cd81-4480-8129-a6f5cd31ab4d","Type":"ContainerStarted","Data":"9686edaed5484423d38379c3ba8b34f6e2ab0138983c4ff26361a9dc3cfca943"}
Oct 30 19:35:59.044150 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:59.043739    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" event={"ID":"6dd4222c-cd81-4480-8129-a6f5cd31ab4d","Type":"ContainerStarted","Data":"274e3b9798c615178f38d5e7933c70345a38ed370d3a3b02aea0c0e5f382257c"}
Oct 30 19:35:59.044150 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:59.043776    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:59.044150 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:59.043827    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:35:59.118754 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:59.118726    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"]
Oct 30 19:35:59.118950 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:59.118927    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" containerName="kserve-container" containerID="cri-o://895d78acde0b504225229ba79fae98d46bbaffde7c58dd69b085195c5b91a29c" gracePeriod=30
Oct 30 19:35:59.119059 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:59.118966    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" containerName="kube-rbac-proxy" containerID="cri-o://be98d418410fc0006e721fac7ec3a0402774438e3eb4e0ad1ef3db456e4aed76" gracePeriod=30
Oct 30 19:35:59.231669 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:59.231612    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" podStartSLOduration=3.308580717 podStartE2EDuration="12.231591813s" totalImagesPullingTime="5.20531656s" totalInitContainerRuntime="3.717694536s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:59.230681579 +0000 UTC m=+3453.748366481" watchObservedRunningTime="2026-10-30 19:35:59.231591813 +0000 UTC m=+3453.749276716"
Oct 30 19:35:59.839338 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:35:59.839295    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.64:8643/healthz\": dial tcp 10.133.0.64:8643: connect: connection refused"
Oct 30 19:36:00.049533 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:00.049500    4511 generic.go:356] "Generic (PLEG): container finished" podID="71ba8320-c44f-41fe-8420-f59be98dc154" containerID="be98d418410fc0006e721fac7ec3a0402774438e3eb4e0ad1ef3db456e4aed76" exitCode=0
Oct 30 19:36:00.049963 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:00.049572    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" event={"ID":"71ba8320-c44f-41fe-8420-f59be98dc154","Type":"ContainerDied","Data":"be98d418410fc0006e721fac7ec3a0402774438e3eb4e0ad1ef3db456e4aed76"}
Oct 30 19:36:00.049963 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:00.049660    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:36:01.960632 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:01.960606    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:36:02.023418 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.023351    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ba8320-c44f-41fe-8420-f59be98dc154-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71ba8320-c44f-41fe-8420-f59be98dc154-cabundle-cert\") pod \"71ba8320-c44f-41fe-8420-f59be98dc154\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") "
Oct 30 19:36:02.023418 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.023391    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ba8320-c44f-41fe-8420-f59be98dc154-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71ba8320-c44f-41fe-8420-f59be98dc154-kserve-provision-location\") pod \"71ba8320-c44f-41fe-8420-f59be98dc154\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") "
Oct 30 19:36:02.023602 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.023421    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ba8320-c44f-41fe-8420-f59be98dc154-kube-api-access-gt6rf\" (UniqueName: \"kubernetes.io/projected/71ba8320-c44f-41fe-8420-f59be98dc154-kube-api-access-gt6rf\") pod \"71ba8320-c44f-41fe-8420-f59be98dc154\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") "
Oct 30 19:36:02.023602 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.023472    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ba8320-c44f-41fe-8420-f59be98dc154-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71ba8320-c44f-41fe-8420-f59be98dc154-kube-rbac-proxy-sar-config\") pod \"71ba8320-c44f-41fe-8420-f59be98dc154\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") "
Oct 30 19:36:02.023602 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.023512    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71ba8320-c44f-41fe-8420-f59be98dc154-proxy-tls\" (UniqueName: \"kubernetes.io/secret/71ba8320-c44f-41fe-8420-f59be98dc154-proxy-tls\") pod \"71ba8320-c44f-41fe-8420-f59be98dc154\" (UID: \"71ba8320-c44f-41fe-8420-f59be98dc154\") "
Oct 30 19:36:02.023800 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.023773    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ba8320-c44f-41fe-8420-f59be98dc154-cabundle-cert" pod "71ba8320-c44f-41fe-8420-f59be98dc154" (UID: "71ba8320-c44f-41fe-8420-f59be98dc154"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:02.023800 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.023790    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ba8320-c44f-41fe-8420-f59be98dc154-kube-rbac-proxy-sar-config" pod "71ba8320-c44f-41fe-8420-f59be98dc154" (UID: "71ba8320-c44f-41fe-8420-f59be98dc154"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:02.025606 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.025573    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ba8320-c44f-41fe-8420-f59be98dc154-kube-api-access-gt6rf" pod "71ba8320-c44f-41fe-8420-f59be98dc154" (UID: "71ba8320-c44f-41fe-8420-f59be98dc154"). InnerVolumeSpecName "kube-api-access-gt6rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:02.025716 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.025615    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71ba8320-c44f-41fe-8420-f59be98dc154-proxy-tls" pod "71ba8320-c44f-41fe-8420-f59be98dc154" (UID: "71ba8320-c44f-41fe-8420-f59be98dc154"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:02.034194 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.034166    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ba8320-c44f-41fe-8420-f59be98dc154-kserve-provision-location" pod "71ba8320-c44f-41fe-8420-f59be98dc154" (UID: "71ba8320-c44f-41fe-8420-f59be98dc154"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:02.058805 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.058769    4511 generic.go:356] "Generic (PLEG): container finished" podID="71ba8320-c44f-41fe-8420-f59be98dc154" containerID="895d78acde0b504225229ba79fae98d46bbaffde7c58dd69b085195c5b91a29c" exitCode=0
Oct 30 19:36:02.058897 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.058837    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"
Oct 30 19:36:02.058897 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.058861    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" event={"ID":"71ba8320-c44f-41fe-8420-f59be98dc154","Type":"ContainerDied","Data":"895d78acde0b504225229ba79fae98d46bbaffde7c58dd69b085195c5b91a29c"}
Oct 30 19:36:02.058978 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.058930    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm" event={"ID":"71ba8320-c44f-41fe-8420-f59be98dc154","Type":"ContainerDied","Data":"cb0525802d6246ef951da1e6f729a943e13bb57170fdeb8d178f96869fa996b6"}
Oct 30 19:36:02.058978 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.058969    4511 scope.go:122] "RemoveContainer" containerID="be98d418410fc0006e721fac7ec3a0402774438e3eb4e0ad1ef3db456e4aed76"
Oct 30 19:36:02.067578 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.067563    4511 scope.go:122] "RemoveContainer" containerID="895d78acde0b504225229ba79fae98d46bbaffde7c58dd69b085195c5b91a29c"
Oct 30 19:36:02.074622 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.074608    4511 scope.go:122] "RemoveContainer" containerID="519b5414e78bdec6b8a815f001e04ffbf1250aa09ec27d8765f42ea8ba6e2b45"
Oct 30 19:36:02.081280 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.081263    4511 scope.go:122] "RemoveContainer" containerID="be98d418410fc0006e721fac7ec3a0402774438e3eb4e0ad1ef3db456e4aed76"
Oct 30 19:36:02.081539 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:36:02.081521    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be98d418410fc0006e721fac7ec3a0402774438e3eb4e0ad1ef3db456e4aed76\": container with ID starting with be98d418410fc0006e721fac7ec3a0402774438e3eb4e0ad1ef3db456e4aed76 not found: ID does not exist" containerID="be98d418410fc0006e721fac7ec3a0402774438e3eb4e0ad1ef3db456e4aed76"
Oct 30 19:36:02.081590 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.081549    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be98d418410fc0006e721fac7ec3a0402774438e3eb4e0ad1ef3db456e4aed76"} err="failed to get container status \"be98d418410fc0006e721fac7ec3a0402774438e3eb4e0ad1ef3db456e4aed76\": rpc error: code = NotFound desc = could not find container \"be98d418410fc0006e721fac7ec3a0402774438e3eb4e0ad1ef3db456e4aed76\": container with ID starting with be98d418410fc0006e721fac7ec3a0402774438e3eb4e0ad1ef3db456e4aed76 not found: ID does not exist"
Oct 30 19:36:02.081590 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.081565    4511 scope.go:122] "RemoveContainer" containerID="895d78acde0b504225229ba79fae98d46bbaffde7c58dd69b085195c5b91a29c"
Oct 30 19:36:02.081789 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:36:02.081772    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"895d78acde0b504225229ba79fae98d46bbaffde7c58dd69b085195c5b91a29c\": container with ID starting with 895d78acde0b504225229ba79fae98d46bbaffde7c58dd69b085195c5b91a29c not found: ID does not exist" containerID="895d78acde0b504225229ba79fae98d46bbaffde7c58dd69b085195c5b91a29c"
Oct 30 19:36:02.081858 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.081799    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"895d78acde0b504225229ba79fae98d46bbaffde7c58dd69b085195c5b91a29c"} err="failed to get container status \"895d78acde0b504225229ba79fae98d46bbaffde7c58dd69b085195c5b91a29c\": rpc error: code = NotFound desc = could not find container \"895d78acde0b504225229ba79fae98d46bbaffde7c58dd69b085195c5b91a29c\": container with ID starting with 895d78acde0b504225229ba79fae98d46bbaffde7c58dd69b085195c5b91a29c not found: ID does not exist"
Oct 30 19:36:02.081858 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.081820    4511 scope.go:122] "RemoveContainer" containerID="519b5414e78bdec6b8a815f001e04ffbf1250aa09ec27d8765f42ea8ba6e2b45"
Oct 30 19:36:02.082062 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:36:02.082044    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519b5414e78bdec6b8a815f001e04ffbf1250aa09ec27d8765f42ea8ba6e2b45\": container with ID starting with 519b5414e78bdec6b8a815f001e04ffbf1250aa09ec27d8765f42ea8ba6e2b45 not found: ID does not exist" containerID="519b5414e78bdec6b8a815f001e04ffbf1250aa09ec27d8765f42ea8ba6e2b45"
Oct 30 19:36:02.082096 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.082066    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519b5414e78bdec6b8a815f001e04ffbf1250aa09ec27d8765f42ea8ba6e2b45"} err="failed to get container status \"519b5414e78bdec6b8a815f001e04ffbf1250aa09ec27d8765f42ea8ba6e2b45\": rpc error: code = NotFound desc = could not find container \"519b5414e78bdec6b8a815f001e04ffbf1250aa09ec27d8765f42ea8ba6e2b45\": container with ID starting with 519b5414e78bdec6b8a815f001e04ffbf1250aa09ec27d8765f42ea8ba6e2b45 not found: ID does not exist"
Oct 30 19:36:02.124349 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.124330    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71ba8320-c44f-41fe-8420-f59be98dc154-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:02.124453 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.124354    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71ba8320-c44f-41fe-8420-f59be98dc154-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:02.124453 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.124370    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt6rf\" (UniqueName: \"kubernetes.io/projected/71ba8320-c44f-41fe-8420-f59be98dc154-kube-api-access-gt6rf\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:02.124453 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.124382    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71ba8320-c44f-41fe-8420-f59be98dc154-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:02.124453 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.124396    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71ba8320-c44f-41fe-8420-f59be98dc154-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:02.236234 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.236204    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"]
Oct 30 19:36:02.240797 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:02.240774    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-paddle-runtime-predictor-6f4576b95c-fcbnm"]
Oct 30 19:36:03.656463 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:03.656431    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" path="/var/lib/kubelet/pods/71ba8320-c44f-41fe-8420-f59be98dc154/volumes"
Oct 30 19:36:06.058333 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:06.058306    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:36:10.155813 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:36:10.155773    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:36:26.059238 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:26.059200    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.66:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.66:8080: connect: connection refused"
Oct 30 19:36:28.218637 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:28.218591    4511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.66:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.66:8080: connect: connection refused"
Oct 30 19:36:36.225503 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:36:36.225476    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:36:40.157843 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:36:40.157805    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:37:06.226984 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:06.226955    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:37:10.156004 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:37:10.155963    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:37:15.143016 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.142983    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"]
Oct 30 19:37:15.143417 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.143318    4511 cpu_manager.go:414] "containerMap: removing container" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" containerName="kube-rbac-proxy"
Oct 30 19:37:15.143417 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.143330    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" containerName="kube-rbac-proxy"
Oct 30 19:37:15.143417 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.143350    4511 cpu_manager.go:414] "containerMap: removing container" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" containerName="storage-initializer"
Oct 30 19:37:15.143417 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.143355    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" containerName="storage-initializer"
Oct 30 19:37:15.143417 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.143365    4511 cpu_manager.go:414] "containerMap: removing container" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" containerName="kserve-container"
Oct 30 19:37:15.143417 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.143369    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" containerName="kserve-container"
Oct 30 19:37:15.143607 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.143428    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" containerName="kube-rbac-proxy"
Oct 30 19:37:15.143607 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.143439    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ba8320-c44f-41fe-8420-f59be98dc154" containerName="kserve-container"
Oct 30 19:37:15.146518 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.146500    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:15.158247 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.158213    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h964m\" (UniqueName: \"kubernetes.io/projected/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kube-api-access-h964m\") pod \"isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") " pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:15.158368 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.158283    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") " pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:15.158368 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.158320    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-proxy-tls\") pod \"isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") " pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:15.158496 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.158383    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") " pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:15.158617 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.158591    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") " pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:15.197620 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.197592    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"openshift-service-ca.crt\""
Oct 30 19:37:15.197620 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.197592    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Oct 30 19:37:15.244948 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.244918    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:37:15.245044 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.244977    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"seaweedfs-s3-creds\""
Oct 30 19:37:15.245044 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.245029    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Oct 30 19:37:15.245169 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.245107    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-4a3552942a\"/\"kube-root-ca.crt\""
Oct 30 19:37:15.245218 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.245193    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-4a3552942a\"/\"default-dockercfg-8gv6n\""
Oct 30 19:37:15.249219 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.249198    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"]
Oct 30 19:37:15.259346 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.259320    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h964m\" (UniqueName: \"kubernetes.io/projected/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kube-api-access-h964m\") pod \"isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") " pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:15.259445 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.259376    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") " pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:15.259445 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.259408    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-proxy-tls\") pod \"isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") " pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:15.259445 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.259429    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") " pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:15.259609 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.259449    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") " pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:15.259933 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.259910    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") " pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:15.260259 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.260237    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") " pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:15.260367 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.260241    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") " pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:15.262121 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.262103    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-proxy-tls\") pod \"isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") " pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:15.319333 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.319293    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h964m\" (UniqueName: \"kubernetes.io/projected/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kube-api-access-h964m\") pod \"isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") " pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:15.468779 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.468691    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:15.891700 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:15.891667    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"]
Oct 30 19:37:15.896040 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:37:15.896014    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9ce467_d2be_48d0_b134_5f6aa7a8259a.slice/crio-23c3291e2776a6913a3ab9fda7adb9339acfd5db6871ec55e1a5cfc129c0d1f8 WatchSource:0}: Error finding container 23c3291e2776a6913a3ab9fda7adb9339acfd5db6871ec55e1a5cfc129c0d1f8: Status 404 returned error can't find the container with id 23c3291e2776a6913a3ab9fda7adb9339acfd5db6871ec55e1a5cfc129c0d1f8
Oct 30 19:37:16.345956 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:16.345923    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" event={"ID":"fe9ce467-d2be-48d0-b134-5f6aa7a8259a","Type":"ContainerStarted","Data":"e9d5e8757f77c298ae4f3a859f61975c6a60ed67c0892f81344ffce9ebb95593"}
Oct 30 19:37:16.346347 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:16.346000    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" event={"ID":"fe9ce467-d2be-48d0-b134-5f6aa7a8259a","Type":"ContainerStarted","Data":"23c3291e2776a6913a3ab9fda7adb9339acfd5db6871ec55e1a5cfc129c0d1f8"}
Oct 30 19:37:18.019096 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:18.019066    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"]
Oct 30 19:37:18.019532 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:18.019317    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kserve-container" containerID="cri-o://7869af4924ee0d41d2ebc9930eab236389d958602b8b987d2da03317878331fe" gracePeriod=30
Oct 30 19:37:18.019532 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:18.019368    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kserve-agent" containerID="cri-o://274e3b9798c615178f38d5e7933c70345a38ed370d3a3b02aea0c0e5f382257c" gracePeriod=30
Oct 30 19:37:18.019532 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:18.019377    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kube-rbac-proxy" containerID="cri-o://9686edaed5484423d38379c3ba8b34f6e2ab0138983c4ff26361a9dc3cfca943" gracePeriod=30
Oct 30 19:37:18.356588 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:18.356559    4511 generic.go:356] "Generic (PLEG): container finished" podID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerID="9686edaed5484423d38379c3ba8b34f6e2ab0138983c4ff26361a9dc3cfca943" exitCode=0
Oct 30 19:37:18.356758 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:18.356627    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" event={"ID":"6dd4222c-cd81-4480-8129-a6f5cd31ab4d","Type":"ContainerDied","Data":"9686edaed5484423d38379c3ba8b34f6e2ab0138983c4ff26361a9dc3cfca943"}
Oct 30 19:37:20.366902 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:20.366867    4511 generic.go:356] "Generic (PLEG): container finished" podID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerID="7869af4924ee0d41d2ebc9930eab236389d958602b8b987d2da03317878331fe" exitCode=0
Oct 30 19:37:20.367347 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:20.366944    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" event={"ID":"6dd4222c-cd81-4480-8129-a6f5cd31ab4d","Type":"ContainerDied","Data":"7869af4924ee0d41d2ebc9930eab236389d958602b8b987d2da03317878331fe"}
Oct 30 19:37:20.368469 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:20.368449    4511 generic.go:356] "Generic (PLEG): container finished" podID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerID="e9d5e8757f77c298ae4f3a859f61975c6a60ed67c0892f81344ffce9ebb95593" exitCode=0
Oct 30 19:37:20.368562 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:20.368501    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" event={"ID":"fe9ce467-d2be-48d0-b134-5f6aa7a8259a","Type":"ContainerDied","Data":"e9d5e8757f77c298ae4f3a859f61975c6a60ed67c0892f81344ffce9ebb95593"}
Oct 30 19:37:21.054331 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:21.054291    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.66:8643/healthz\": dial tcp 10.133.0.66:8643: connect: connection refused"
Oct 30 19:37:21.374742 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:21.374669    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" event={"ID":"fe9ce467-d2be-48d0-b134-5f6aa7a8259a","Type":"ContainerStarted","Data":"82abfc3072757b2d0361f8c8e23da0c0e7315915a35c4d7ab730bd267ce05ee8"}
Oct 30 19:37:21.374742 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:21.374727    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" event={"ID":"fe9ce467-d2be-48d0-b134-5f6aa7a8259a","Type":"ContainerStarted","Data":"892fdf5561317a8a3b46f7fdbf390aacf08f608cbc832e5276009180bcbb3e93"}
Oct 30 19:37:21.375149 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:21.374915    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:21.491945 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:21.491893    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" podStartSLOduration=2.529692172 podStartE2EDuration="6.491878973s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.962186801s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:21.490386098 +0000 UTC m=+3536.008070998" watchObservedRunningTime="2026-10-30 19:37:21.491878973 +0000 UTC m=+3536.009563870"
Oct 30 19:37:22.378622 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:22.378593    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:22.379544 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:22.379514    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 30 19:37:23.383292 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:23.383255    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 30 19:37:26.054005 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:26.053956    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.66:8643/healthz\": dial tcp 10.133.0.66:8643: connect: connection refused"
Oct 30 19:37:26.059364 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:26.059338    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.66:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.66:8080: connect: connection refused"
Oct 30 19:37:28.388470 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:28.388444    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:37:28.388931 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:28.388907    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 30 19:37:31.054210 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:31.054174    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.66:8643/healthz\": dial tcp 10.133.0.66:8643: connect: connection refused"
Oct 30 19:37:31.054647 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:31.054299    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:37:36.054508 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:36.054470    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.66:8643/healthz\": dial tcp 10.133.0.66:8643: connect: connection refused"
Oct 30 19:37:36.058867 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:36.058839    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.66:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.66:8080: connect: connection refused"
Oct 30 19:37:38.389751 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:38.389711    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 30 19:37:40.156981 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:37:40.156929    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:37:41.053706 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:41.053663    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.66:8643/healthz\": dial tcp 10.133.0.66:8643: connect: connection refused"
Oct 30 19:37:46.053861 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:46.053824    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.66:8643/healthz\": dial tcp 10.133.0.66:8643: connect: connection refused"
Oct 30 19:37:46.059215 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:46.059187    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.66:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.66:8080: connect: connection refused"
Oct 30 19:37:46.059331 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:46.059317    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:37:48.159748 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.159723    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:37:48.310619 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.310586    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kserve-provision-location\") pod \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") "
Oct 30 19:37:48.310787 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.310623    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-proxy-tls\") pod \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") "
Oct 30 19:37:48.310787 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.310652    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-cabundle-cert\") pod \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") "
Oct 30 19:37:48.310787 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.310724    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kube-api-access-lqrhx\" (UniqueName: \"kubernetes.io/projected/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kube-api-access-lqrhx\") pod \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") "
Oct 30 19:37:48.310787 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.310762    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kube-rbac-proxy-sar-config\") pod \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\" (UID: \"6dd4222c-cd81-4480-8129-a6f5cd31ab4d\") "
Oct 30 19:37:48.311004 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.310948    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kserve-provision-location" pod "6dd4222c-cd81-4480-8129-a6f5cd31ab4d" (UID: "6dd4222c-cd81-4480-8129-a6f5cd31ab4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:48.311064 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.311038    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:48.311202 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.311175    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-cabundle-cert" pod "6dd4222c-cd81-4480-8129-a6f5cd31ab4d" (UID: "6dd4222c-cd81-4480-8129-a6f5cd31ab4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:48.311352 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.311326    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kube-rbac-proxy-sar-config" pod "6dd4222c-cd81-4480-8129-a6f5cd31ab4d" (UID: "6dd4222c-cd81-4480-8129-a6f5cd31ab4d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:48.312866 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.312843    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-proxy-tls" pod "6dd4222c-cd81-4480-8129-a6f5cd31ab4d" (UID: "6dd4222c-cd81-4480-8129-a6f5cd31ab4d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:48.313085 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.313064    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kube-api-access-lqrhx" pod "6dd4222c-cd81-4480-8129-a6f5cd31ab4d" (UID: "6dd4222c-cd81-4480-8129-a6f5cd31ab4d"). InnerVolumeSpecName "kube-api-access-lqrhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:48.389263 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.389224    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 30 19:37:48.412464 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.412439    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqrhx\" (UniqueName: \"kubernetes.io/projected/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kube-api-access-lqrhx\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:48.412464 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.412461    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:48.412617 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.412474    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:48.412617 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.412488    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dd4222c-cd81-4480-8129-a6f5cd31ab4d-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:48.483302 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.483277    4511 generic.go:356] "Generic (PLEG): container finished" podID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerID="274e3b9798c615178f38d5e7933c70345a38ed370d3a3b02aea0c0e5f382257c" exitCode=0
Oct 30 19:37:48.483411 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.483331    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" event={"ID":"6dd4222c-cd81-4480-8129-a6f5cd31ab4d","Type":"ContainerDied","Data":"274e3b9798c615178f38d5e7933c70345a38ed370d3a3b02aea0c0e5f382257c"}
Oct 30 19:37:48.483411 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.483345    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"
Oct 30 19:37:48.483411 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.483386    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6" event={"ID":"6dd4222c-cd81-4480-8129-a6f5cd31ab4d","Type":"ContainerDied","Data":"d471609811ea7a883be224ef8c9886d513cbbd4a9a21a68331b9ada0c30b9913"}
Oct 30 19:37:48.483512 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.483422    4511 scope.go:122] "RemoveContainer" containerID="9686edaed5484423d38379c3ba8b34f6e2ab0138983c4ff26361a9dc3cfca943"
Oct 30 19:37:48.491904 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.491885    4511 scope.go:122] "RemoveContainer" containerID="274e3b9798c615178f38d5e7933c70345a38ed370d3a3b02aea0c0e5f382257c"
Oct 30 19:37:48.498876 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.498854    4511 scope.go:122] "RemoveContainer" containerID="7869af4924ee0d41d2ebc9930eab236389d958602b8b987d2da03317878331fe"
Oct 30 19:37:48.505658 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.505641    4511 scope.go:122] "RemoveContainer" containerID="64442bb77921246e6a4e1b77d1dd1d46d45de4c382ee77c164fd5312a5965632"
Oct 30 19:37:48.512219 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.512201    4511 scope.go:122] "RemoveContainer" containerID="9686edaed5484423d38379c3ba8b34f6e2ab0138983c4ff26361a9dc3cfca943"
Oct 30 19:37:48.512480 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:37:48.512462    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9686edaed5484423d38379c3ba8b34f6e2ab0138983c4ff26361a9dc3cfca943\": container with ID starting with 9686edaed5484423d38379c3ba8b34f6e2ab0138983c4ff26361a9dc3cfca943 not found: ID does not exist" containerID="9686edaed5484423d38379c3ba8b34f6e2ab0138983c4ff26361a9dc3cfca943"
Oct 30 19:37:48.512556 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.512492    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9686edaed5484423d38379c3ba8b34f6e2ab0138983c4ff26361a9dc3cfca943"} err="failed to get container status \"9686edaed5484423d38379c3ba8b34f6e2ab0138983c4ff26361a9dc3cfca943\": rpc error: code = NotFound desc = could not find container \"9686edaed5484423d38379c3ba8b34f6e2ab0138983c4ff26361a9dc3cfca943\": container with ID starting with 9686edaed5484423d38379c3ba8b34f6e2ab0138983c4ff26361a9dc3cfca943 not found: ID does not exist"
Oct 30 19:37:48.512556 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.512516    4511 scope.go:122] "RemoveContainer" containerID="274e3b9798c615178f38d5e7933c70345a38ed370d3a3b02aea0c0e5f382257c"
Oct 30 19:37:48.512762 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:37:48.512746    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"274e3b9798c615178f38d5e7933c70345a38ed370d3a3b02aea0c0e5f382257c\": container with ID starting with 274e3b9798c615178f38d5e7933c70345a38ed370d3a3b02aea0c0e5f382257c not found: ID does not exist" containerID="274e3b9798c615178f38d5e7933c70345a38ed370d3a3b02aea0c0e5f382257c"
Oct 30 19:37:48.512828 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.512770    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"274e3b9798c615178f38d5e7933c70345a38ed370d3a3b02aea0c0e5f382257c"} err="failed to get container status \"274e3b9798c615178f38d5e7933c70345a38ed370d3a3b02aea0c0e5f382257c\": rpc error: code = NotFound desc = could not find container \"274e3b9798c615178f38d5e7933c70345a38ed370d3a3b02aea0c0e5f382257c\": container with ID starting with 274e3b9798c615178f38d5e7933c70345a38ed370d3a3b02aea0c0e5f382257c not found: ID does not exist"
Oct 30 19:37:48.512828 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.512789    4511 scope.go:122] "RemoveContainer" containerID="7869af4924ee0d41d2ebc9930eab236389d958602b8b987d2da03317878331fe"
Oct 30 19:37:48.513017 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:37:48.512998    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7869af4924ee0d41d2ebc9930eab236389d958602b8b987d2da03317878331fe\": container with ID starting with 7869af4924ee0d41d2ebc9930eab236389d958602b8b987d2da03317878331fe not found: ID does not exist" containerID="7869af4924ee0d41d2ebc9930eab236389d958602b8b987d2da03317878331fe"
Oct 30 19:37:48.513058 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.513026    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7869af4924ee0d41d2ebc9930eab236389d958602b8b987d2da03317878331fe"} err="failed to get container status \"7869af4924ee0d41d2ebc9930eab236389d958602b8b987d2da03317878331fe\": rpc error: code = NotFound desc = could not find container \"7869af4924ee0d41d2ebc9930eab236389d958602b8b987d2da03317878331fe\": container with ID starting with 7869af4924ee0d41d2ebc9930eab236389d958602b8b987d2da03317878331fe not found: ID does not exist"
Oct 30 19:37:48.513058 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.513043    4511 scope.go:122] "RemoveContainer" containerID="64442bb77921246e6a4e1b77d1dd1d46d45de4c382ee77c164fd5312a5965632"
Oct 30 19:37:48.513544 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:37:48.513529    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64442bb77921246e6a4e1b77d1dd1d46d45de4c382ee77c164fd5312a5965632\": container with ID starting with 64442bb77921246e6a4e1b77d1dd1d46d45de4c382ee77c164fd5312a5965632 not found: ID does not exist" containerID="64442bb77921246e6a4e1b77d1dd1d46d45de4c382ee77c164fd5312a5965632"
Oct 30 19:37:48.513595 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.513550    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64442bb77921246e6a4e1b77d1dd1d46d45de4c382ee77c164fd5312a5965632"} err="failed to get container status \"64442bb77921246e6a4e1b77d1dd1d46d45de4c382ee77c164fd5312a5965632\": rpc error: code = NotFound desc = could not find container \"64442bb77921246e6a4e1b77d1dd1d46d45de4c382ee77c164fd5312a5965632\": container with ID starting with 64442bb77921246e6a4e1b77d1dd1d46d45de4c382ee77c164fd5312a5965632 not found: ID does not exist"
Oct 30 19:37:48.652800 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.652739    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"]
Oct 30 19:37:48.657065 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:48.657045    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-sklearn-mcp-predictor-56b5dcd664-wlww6"]
Oct 30 19:37:49.655910 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:49.655874    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" path="/var/lib/kubelet/pods/6dd4222c-cd81-4480-8129-a6f5cd31ab4d/volumes"
Oct 30 19:37:51.352316 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.352286    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"]
Oct 30 19:37:51.352678 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.352607    4511 cpu_manager.go:414] "containerMap: removing container" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="storage-initializer"
Oct 30 19:37:51.352678 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.352618    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="storage-initializer"
Oct 30 19:37:51.352678 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.352629    4511 cpu_manager.go:414] "containerMap: removing container" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kserve-container"
Oct 30 19:37:51.352678 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.352637    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kserve-container"
Oct 30 19:37:51.352678 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.352655    4511 cpu_manager.go:414] "containerMap: removing container" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kserve-agent"
Oct 30 19:37:51.352678 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.352660    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kserve-agent"
Oct 30 19:37:51.352678 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.352669    4511 cpu_manager.go:414] "containerMap: removing container" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kube-rbac-proxy"
Oct 30 19:37:51.352678 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.352673    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kube-rbac-proxy"
Oct 30 19:37:51.352926 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.352729    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kube-rbac-proxy"
Oct 30 19:37:51.352926 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.352740    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kserve-container"
Oct 30 19:37:51.352926 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.352746    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd4222c-cd81-4480-8129-a6f5cd31ab4d" containerName="kserve-agent"
Oct 30 19:37:51.357550 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.357533    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:37:51.408310 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.408281    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Oct 30 19:37:51.408424 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.408354    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"isvc-pmml-predictor-serving-cert\""
Oct 30 19:37:51.408424 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.408370    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"seaweedfs-s3-creds\""
Oct 30 19:37:51.432270 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.432244    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vbvv\" (UniqueName: \"kubernetes.io/projected/216ed3b4-58db-4a73-bd92-326baa7b9aff-kube-api-access-6vbvv\") pod \"isvc-pmml-predictor-8f5ffdfb5-c674t\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") " pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:37:51.432379 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.432275    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/216ed3b4-58db-4a73-bd92-326baa7b9aff-proxy-tls\") pod \"isvc-pmml-predictor-8f5ffdfb5-c674t\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") " pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:37:51.432379 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.432319    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/216ed3b4-58db-4a73-bd92-326baa7b9aff-kserve-provision-location\") pod \"isvc-pmml-predictor-8f5ffdfb5-c674t\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") " pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:37:51.432461 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.432436    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/216ed3b4-58db-4a73-bd92-326baa7b9aff-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8f5ffdfb5-c674t\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") " pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:37:51.432498 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.432473    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/216ed3b4-58db-4a73-bd92-326baa7b9aff-cabundle-cert\") pod \"isvc-pmml-predictor-8f5ffdfb5-c674t\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") " pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:37:51.455468 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.455444    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"kube-root-ca.crt\""
Oct 30 19:37:51.455544 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.455508    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"openshift-service-ca.crt\""
Oct 30 19:37:51.455584 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.455548    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-6eaea25a75\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:37:51.455616 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.455597    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-6eaea25a75\"/\"default-dockercfg-29p4g\""
Oct 30 19:37:51.459725 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.459706    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"]
Oct 30 19:37:51.532946 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.532913    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vbvv\" (UniqueName: \"kubernetes.io/projected/216ed3b4-58db-4a73-bd92-326baa7b9aff-kube-api-access-6vbvv\") pod \"isvc-pmml-predictor-8f5ffdfb5-c674t\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") " pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:37:51.532946 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.532947    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/216ed3b4-58db-4a73-bd92-326baa7b9aff-proxy-tls\") pod \"isvc-pmml-predictor-8f5ffdfb5-c674t\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") " pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:37:51.533174 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.532983    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/216ed3b4-58db-4a73-bd92-326baa7b9aff-kserve-provision-location\") pod \"isvc-pmml-predictor-8f5ffdfb5-c674t\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") " pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:37:51.533174 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.533017    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/216ed3b4-58db-4a73-bd92-326baa7b9aff-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8f5ffdfb5-c674t\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") " pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:37:51.533174 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.533042    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/216ed3b4-58db-4a73-bd92-326baa7b9aff-cabundle-cert\") pod \"isvc-pmml-predictor-8f5ffdfb5-c674t\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") " pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:37:51.533474 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.533456    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/216ed3b4-58db-4a73-bd92-326baa7b9aff-kserve-provision-location\") pod \"isvc-pmml-predictor-8f5ffdfb5-c674t\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") " pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:37:51.533870 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.533848    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/216ed3b4-58db-4a73-bd92-326baa7b9aff-cabundle-cert\") pod \"isvc-pmml-predictor-8f5ffdfb5-c674t\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") " pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:37:51.533870 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.533848    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/216ed3b4-58db-4a73-bd92-326baa7b9aff-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8f5ffdfb5-c674t\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") " pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:37:51.535635 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.535612    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/216ed3b4-58db-4a73-bd92-326baa7b9aff-proxy-tls\") pod \"isvc-pmml-predictor-8f5ffdfb5-c674t\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") " pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:37:51.591895 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.591864    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vbvv\" (UniqueName: \"kubernetes.io/projected/216ed3b4-58db-4a73-bd92-326baa7b9aff-kube-api-access-6vbvv\") pod \"isvc-pmml-predictor-8f5ffdfb5-c674t\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") " pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:37:51.673644 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:51.673568    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:37:52.100671 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:52.100642    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"]
Oct 30 19:37:52.107102 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:37:52.107077    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216ed3b4_58db_4a73_bd92_326baa7b9aff.slice/crio-8a35908ed3d77292622761146de0e5bd2da80c9a5724fccf0f4aa542cd9df7f8 WatchSource:0}: Error finding container 8a35908ed3d77292622761146de0e5bd2da80c9a5724fccf0f4aa542cd9df7f8: Status 404 returned error can't find the container with id 8a35908ed3d77292622761146de0e5bd2da80c9a5724fccf0f4aa542cd9df7f8
Oct 30 19:37:52.503174 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:52.503065    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" event={"ID":"216ed3b4-58db-4a73-bd92-326baa7b9aff","Type":"ContainerStarted","Data":"215d3d0a4ac8b6f1f15b87e55e920af9c01a0456368d191db27a9adfbedb2845"}
Oct 30 19:37:52.503174 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:52.503154    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" event={"ID":"216ed3b4-58db-4a73-bd92-326baa7b9aff","Type":"ContainerStarted","Data":"8a35908ed3d77292622761146de0e5bd2da80c9a5724fccf0f4aa542cd9df7f8"}
Oct 30 19:37:56.519030 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:56.518986    4511 generic.go:356] "Generic (PLEG): container finished" podID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerID="215d3d0a4ac8b6f1f15b87e55e920af9c01a0456368d191db27a9adfbedb2845" exitCode=0
Oct 30 19:37:56.519463 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:56.519068    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" event={"ID":"216ed3b4-58db-4a73-bd92-326baa7b9aff","Type":"ContainerDied","Data":"215d3d0a4ac8b6f1f15b87e55e920af9c01a0456368d191db27a9adfbedb2845"}
Oct 30 19:37:57.525946 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:57.525911    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" event={"ID":"216ed3b4-58db-4a73-bd92-326baa7b9aff","Type":"ContainerStarted","Data":"73a510a4a809f4dc2364e67e26988b77f5e893d152d229a68f9fa37817d3074d"}
Oct 30 19:37:57.526575 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:57.525985    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" event={"ID":"216ed3b4-58db-4a73-bd92-326baa7b9aff","Type":"ContainerStarted","Data":"97699b08a6ae61dac4d482dd1bad3be6ae4ccc600af7ee2833d80c997cdb4990"}
Oct 30 19:37:57.641955 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:57.641906    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" podStartSLOduration=3.068915427 podStartE2EDuration="6.641893203s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.572977776s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:57.641089682 +0000 UTC m=+3572.158774601" watchObservedRunningTime="2026-10-30 19:37:57.641893203 +0000 UTC m=+3572.159578102"
Oct 30 19:37:58.389592 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:37:58.389555    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 30 19:38:02.527294 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:02.527254    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:38:02.527760 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:02.527737    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:38:02.528867 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:02.528841    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 30 19:38:02.532267 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:02.532246    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:38:02.550151 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:02.550101    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 30 19:38:03.554072 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:03.554027    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 30 19:38:08.389667 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:08.389621    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 30 19:38:10.156580 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:38:10.156534    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:38:13.554629 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:13.554593    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 30 19:38:18.389101 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:18.389056    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 30 19:38:23.554242 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:23.554200    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 30 19:38:25.912576 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:25.912474    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 19:38:25.924339 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:25.924318    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5qjv_2a647480-88ca-47a2-9377-2841ce2e14f9/ovn-acl-logging/0.log"
Oct 30 19:38:28.389799 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:28.389751    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 30 19:38:30.651409 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:30.651367    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 30 19:38:33.554653 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:33.554612    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 30 19:38:40.157367 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:38:40.157330    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:38:40.651770 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:40.651734    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 30 19:38:43.554469 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:43.554427    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 30 19:38:50.651953 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:50.651924    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:38:53.554178 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:53.554112    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 30 19:38:56.949063 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:56.949034    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"]
Oct 30 19:38:56.949560 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:56.949291    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kserve-container" containerID="cri-o://892fdf5561317a8a3b46f7fdbf390aacf08f608cbc832e5276009180bcbb3e93" gracePeriod=30
Oct 30 19:38:56.949560 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:56.949328    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kube-rbac-proxy" containerID="cri-o://82abfc3072757b2d0361f8c8e23da0c0e7315915a35c4d7ab730bd267ce05ee8" gracePeriod=30
Oct 30 19:38:57.769868 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:57.769835    4511 generic.go:356] "Generic (PLEG): container finished" podID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerID="82abfc3072757b2d0361f8c8e23da0c0e7315915a35c4d7ab730bd267ce05ee8" exitCode=0
Oct 30 19:38:57.770055 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:57.769906    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" event={"ID":"fe9ce467-d2be-48d0-b134-5f6aa7a8259a","Type":"ContainerDied","Data":"82abfc3072757b2d0361f8c8e23da0c0e7315915a35c4d7ab730bd267ce05ee8"}
Oct 30 19:38:58.383879 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:38:58.383842    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.67:8643/healthz\": dial tcp 10.133.0.67:8643: connect: connection refused"
Oct 30 19:39:00.652276 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:00.652224    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Oct 30 19:39:00.784600 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:00.784570    4511 generic.go:356] "Generic (PLEG): container finished" podID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerID="892fdf5561317a8a3b46f7fdbf390aacf08f608cbc832e5276009180bcbb3e93" exitCode=0
Oct 30 19:39:00.784742 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:00.784619    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" event={"ID":"fe9ce467-d2be-48d0-b134-5f6aa7a8259a","Type":"ContainerDied","Data":"892fdf5561317a8a3b46f7fdbf390aacf08f608cbc832e5276009180bcbb3e93"}
Oct 30 19:39:00.888910 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:00.888888    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:39:00.948676 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:00.948619    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kube-rbac-proxy-sar-config\") pod \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") "
Oct 30 19:39:00.948676 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:00.948667    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kube-api-access-h964m\" (UniqueName: \"kubernetes.io/projected/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kube-api-access-h964m\") pod \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") "
Oct 30 19:39:00.948817 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:00.948695    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-proxy-tls\") pod \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") "
Oct 30 19:39:00.948817 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:00.948725    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-cabundle-cert\") pod \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") "
Oct 30 19:39:00.948817 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:00.948745    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kserve-provision-location\") pod \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\" (UID: \"fe9ce467-d2be-48d0-b134-5f6aa7a8259a\") "
Oct 30 19:39:00.949170 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:00.949031    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kube-rbac-proxy-sar-config" pod "fe9ce467-d2be-48d0-b134-5f6aa7a8259a" (UID: "fe9ce467-d2be-48d0-b134-5f6aa7a8259a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:00.949170 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:00.949147    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kserve-provision-location" pod "fe9ce467-d2be-48d0-b134-5f6aa7a8259a" (UID: "fe9ce467-d2be-48d0-b134-5f6aa7a8259a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:00.949309 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:00.949119    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-cabundle-cert" pod "fe9ce467-d2be-48d0-b134-5f6aa7a8259a" (UID: "fe9ce467-d2be-48d0-b134-5f6aa7a8259a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:00.950830 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:00.950803    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kube-api-access-h964m" pod "fe9ce467-d2be-48d0-b134-5f6aa7a8259a" (UID: "fe9ce467-d2be-48d0-b134-5f6aa7a8259a"). InnerVolumeSpecName "kube-api-access-h964m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:00.950830 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:00.950815    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-proxy-tls" pod "fe9ce467-d2be-48d0-b134-5f6aa7a8259a" (UID: "fe9ce467-d2be-48d0-b134-5f6aa7a8259a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:01.050298 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:01.050273    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:01.050298 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:01.050292    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h964m\" (UniqueName: \"kubernetes.io/projected/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kube-api-access-h964m\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:01.050298 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:01.050303    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:01.050467 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:01.050311    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:01.050467 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:01.050320    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe9ce467-d2be-48d0-b134-5f6aa7a8259a-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:01.791083 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:01.791038    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x" event={"ID":"fe9ce467-d2be-48d0-b134-5f6aa7a8259a","Type":"ContainerDied","Data":"23c3291e2776a6913a3ab9fda7adb9339acfd5db6871ec55e1a5cfc129c0d1f8"}
Oct 30 19:39:01.791553 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:01.791057    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"
Oct 30 19:39:01.791553 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:01.791160    4511 scope.go:122] "RemoveContainer" containerID="82abfc3072757b2d0361f8c8e23da0c0e7315915a35c4d7ab730bd267ce05ee8"
Oct 30 19:39:01.799338 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:01.799315    4511 scope.go:122] "RemoveContainer" containerID="892fdf5561317a8a3b46f7fdbf390aacf08f608cbc832e5276009180bcbb3e93"
Oct 30 19:39:01.806645 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:01.806627    4511 scope.go:122] "RemoveContainer" containerID="e9d5e8757f77c298ae4f3a859f61975c6a60ed67c0892f81344ffce9ebb95593"
Oct 30 19:39:01.999866 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:01.999791    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"]
Oct 30 19:39:02.047502 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:02.047470    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-4a3552942a/isvc-pmml-runtime-predictor-5fcfd84d44-s9w2x"]
Oct 30 19:39:03.555022 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:03.554980    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 30 19:39:03.655892 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:03.655860    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" path="/var/lib/kubelet/pods/fe9ce467-d2be-48d0-b134-5f6aa7a8259a/volumes"
Oct 30 19:39:10.156653 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:39:10.156570    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:39:13.554741 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:13.554699    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 30 19:39:23.555078 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:23.555044    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:39:32.939589 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:32.939557    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"]
Oct 30 19:39:32.940069 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:32.939767    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kserve-container" containerID="cri-o://97699b08a6ae61dac4d482dd1bad3be6ae4ccc600af7ee2833d80c997cdb4990" gracePeriod=30
Oct 30 19:39:32.940069 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:32.939801    4511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kube-rbac-proxy" containerID="cri-o://73a510a4a809f4dc2364e67e26988b77f5e893d152d229a68f9fa37817d3074d" gracePeriod=30
Oct 30 19:39:33.554609 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:33.554568    4511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Oct 30 19:39:33.912407 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:33.912327    4511 generic.go:356] "Generic (PLEG): container finished" podID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerID="73a510a4a809f4dc2364e67e26988b77f5e893d152d229a68f9fa37817d3074d" exitCode=0
Oct 30 19:39:33.912407 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:33.912366    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" event={"ID":"216ed3b4-58db-4a73-bd92-326baa7b9aff","Type":"ContainerDied","Data":"73a510a4a809f4dc2364e67e26988b77f5e893d152d229a68f9fa37817d3074d"}
Oct 30 19:39:36.688771 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.688748    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:39:36.816496 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.816472    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/216ed3b4-58db-4a73-bd92-326baa7b9aff-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/216ed3b4-58db-4a73-bd92-326baa7b9aff-cabundle-cert\") pod \"216ed3b4-58db-4a73-bd92-326baa7b9aff\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") "
Oct 30 19:39:36.816642 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.816508    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/216ed3b4-58db-4a73-bd92-326baa7b9aff-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/216ed3b4-58db-4a73-bd92-326baa7b9aff-kserve-provision-location\") pod \"216ed3b4-58db-4a73-bd92-326baa7b9aff\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") "
Oct 30 19:39:36.816642 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.816545    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/216ed3b4-58db-4a73-bd92-326baa7b9aff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/216ed3b4-58db-4a73-bd92-326baa7b9aff-kube-rbac-proxy-sar-config\") pod \"216ed3b4-58db-4a73-bd92-326baa7b9aff\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") "
Oct 30 19:39:36.816642 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.816575    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/216ed3b4-58db-4a73-bd92-326baa7b9aff-proxy-tls\" (UniqueName: \"kubernetes.io/secret/216ed3b4-58db-4a73-bd92-326baa7b9aff-proxy-tls\") pod \"216ed3b4-58db-4a73-bd92-326baa7b9aff\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") "
Oct 30 19:39:36.816642 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.816625    4511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/216ed3b4-58db-4a73-bd92-326baa7b9aff-kube-api-access-6vbvv\" (UniqueName: \"kubernetes.io/projected/216ed3b4-58db-4a73-bd92-326baa7b9aff-kube-api-access-6vbvv\") pod \"216ed3b4-58db-4a73-bd92-326baa7b9aff\" (UID: \"216ed3b4-58db-4a73-bd92-326baa7b9aff\") "
Oct 30 19:39:36.816840 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.816823    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/216ed3b4-58db-4a73-bd92-326baa7b9aff-kserve-provision-location" pod "216ed3b4-58db-4a73-bd92-326baa7b9aff" (UID: "216ed3b4-58db-4a73-bd92-326baa7b9aff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:36.816900 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.816875    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216ed3b4-58db-4a73-bd92-326baa7b9aff-cabundle-cert" pod "216ed3b4-58db-4a73-bd92-326baa7b9aff" (UID: "216ed3b4-58db-4a73-bd92-326baa7b9aff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:36.816943 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.816901    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216ed3b4-58db-4a73-bd92-326baa7b9aff-kube-rbac-proxy-sar-config" pod "216ed3b4-58db-4a73-bd92-326baa7b9aff" (UID: "216ed3b4-58db-4a73-bd92-326baa7b9aff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:36.818744 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.818726    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/216ed3b4-58db-4a73-bd92-326baa7b9aff-proxy-tls" pod "216ed3b4-58db-4a73-bd92-326baa7b9aff" (UID: "216ed3b4-58db-4a73-bd92-326baa7b9aff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:36.818811 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.818773    4511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216ed3b4-58db-4a73-bd92-326baa7b9aff-kube-api-access-6vbvv" pod "216ed3b4-58db-4a73-bd92-326baa7b9aff" (UID: "216ed3b4-58db-4a73-bd92-326baa7b9aff"). InnerVolumeSpecName "kube-api-access-6vbvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:36.917466 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.917443    4511 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/216ed3b4-58db-4a73-bd92-326baa7b9aff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:36.917466 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.917465    4511 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/216ed3b4-58db-4a73-bd92-326baa7b9aff-proxy-tls\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:36.917621 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.917476    4511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vbvv\" (UniqueName: \"kubernetes.io/projected/216ed3b4-58db-4a73-bd92-326baa7b9aff-kube-api-access-6vbvv\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:36.917621 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.917484    4511 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/216ed3b4-58db-4a73-bd92-326baa7b9aff-cabundle-cert\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:36.917621 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.917510    4511 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/216ed3b4-58db-4a73-bd92-326baa7b9aff-kserve-provision-location\") on node \"ip-10-0-135-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:36.926292 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.926266    4511 generic.go:356] "Generic (PLEG): container finished" podID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerID="97699b08a6ae61dac4d482dd1bad3be6ae4ccc600af7ee2833d80c997cdb4990" exitCode=0
Oct 30 19:39:36.926403 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.926327    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" event={"ID":"216ed3b4-58db-4a73-bd92-326baa7b9aff","Type":"ContainerDied","Data":"97699b08a6ae61dac4d482dd1bad3be6ae4ccc600af7ee2833d80c997cdb4990"}
Oct 30 19:39:36.926403 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.926331    4511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"
Oct 30 19:39:36.926403 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.926398    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t" event={"ID":"216ed3b4-58db-4a73-bd92-326baa7b9aff","Type":"ContainerDied","Data":"8a35908ed3d77292622761146de0e5bd2da80c9a5724fccf0f4aa542cd9df7f8"}
Oct 30 19:39:36.926542 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.926423    4511 scope.go:122] "RemoveContainer" containerID="73a510a4a809f4dc2364e67e26988b77f5e893d152d229a68f9fa37817d3074d"
Oct 30 19:39:36.938600 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.938583    4511 scope.go:122] "RemoveContainer" containerID="97699b08a6ae61dac4d482dd1bad3be6ae4ccc600af7ee2833d80c997cdb4990"
Oct 30 19:39:36.945748 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.945730    4511 scope.go:122] "RemoveContainer" containerID="215d3d0a4ac8b6f1f15b87e55e920af9c01a0456368d191db27a9adfbedb2845"
Oct 30 19:39:36.952734 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.952719    4511 scope.go:122] "RemoveContainer" containerID="73a510a4a809f4dc2364e67e26988b77f5e893d152d229a68f9fa37817d3074d"
Oct 30 19:39:36.952973 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:39:36.952957    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a510a4a809f4dc2364e67e26988b77f5e893d152d229a68f9fa37817d3074d\": container with ID starting with 73a510a4a809f4dc2364e67e26988b77f5e893d152d229a68f9fa37817d3074d not found: ID does not exist" containerID="73a510a4a809f4dc2364e67e26988b77f5e893d152d229a68f9fa37817d3074d"
Oct 30 19:39:36.953016 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.952981    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a510a4a809f4dc2364e67e26988b77f5e893d152d229a68f9fa37817d3074d"} err="failed to get container status \"73a510a4a809f4dc2364e67e26988b77f5e893d152d229a68f9fa37817d3074d\": rpc error: code = NotFound desc = could not find container \"73a510a4a809f4dc2364e67e26988b77f5e893d152d229a68f9fa37817d3074d\": container with ID starting with 73a510a4a809f4dc2364e67e26988b77f5e893d152d229a68f9fa37817d3074d not found: ID does not exist"
Oct 30 19:39:36.953016 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.952999    4511 scope.go:122] "RemoveContainer" containerID="97699b08a6ae61dac4d482dd1bad3be6ae4ccc600af7ee2833d80c997cdb4990"
Oct 30 19:39:36.953240 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:39:36.953221    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97699b08a6ae61dac4d482dd1bad3be6ae4ccc600af7ee2833d80c997cdb4990\": container with ID starting with 97699b08a6ae61dac4d482dd1bad3be6ae4ccc600af7ee2833d80c997cdb4990 not found: ID does not exist" containerID="97699b08a6ae61dac4d482dd1bad3be6ae4ccc600af7ee2833d80c997cdb4990"
Oct 30 19:39:36.953317 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.953250    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97699b08a6ae61dac4d482dd1bad3be6ae4ccc600af7ee2833d80c997cdb4990"} err="failed to get container status \"97699b08a6ae61dac4d482dd1bad3be6ae4ccc600af7ee2833d80c997cdb4990\": rpc error: code = NotFound desc = could not find container \"97699b08a6ae61dac4d482dd1bad3be6ae4ccc600af7ee2833d80c997cdb4990\": container with ID starting with 97699b08a6ae61dac4d482dd1bad3be6ae4ccc600af7ee2833d80c997cdb4990 not found: ID does not exist"
Oct 30 19:39:36.953317 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.953272    4511 scope.go:122] "RemoveContainer" containerID="215d3d0a4ac8b6f1f15b87e55e920af9c01a0456368d191db27a9adfbedb2845"
Oct 30 19:39:36.953514 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:39:36.953497    4511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215d3d0a4ac8b6f1f15b87e55e920af9c01a0456368d191db27a9adfbedb2845\": container with ID starting with 215d3d0a4ac8b6f1f15b87e55e920af9c01a0456368d191db27a9adfbedb2845 not found: ID does not exist" containerID="215d3d0a4ac8b6f1f15b87e55e920af9c01a0456368d191db27a9adfbedb2845"
Oct 30 19:39:36.953564 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:36.953518    4511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215d3d0a4ac8b6f1f15b87e55e920af9c01a0456368d191db27a9adfbedb2845"} err="failed to get container status \"215d3d0a4ac8b6f1f15b87e55e920af9c01a0456368d191db27a9adfbedb2845\": rpc error: code = NotFound desc = could not find container \"215d3d0a4ac8b6f1f15b87e55e920af9c01a0456368d191db27a9adfbedb2845\": container with ID starting with 215d3d0a4ac8b6f1f15b87e55e920af9c01a0456368d191db27a9adfbedb2845 not found: ID does not exist"
Oct 30 19:39:37.084758 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:37.084690    4511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"]
Oct 30 19:39:37.089802 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:37.089780    4511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-6eaea25a75/isvc-pmml-predictor-8f5ffdfb5-c674t"]
Oct 30 19:39:37.655847 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:39:37.655807    4511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" path="/var/lib/kubelet/pods/216ed3b4-58db-4a73-bd92-326baa7b9aff/volumes"
Oct 30 19:39:40.157553 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:39:40.157522    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:40:10.156257 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:40:10.156219    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:40:40.157026 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:40:40.156941    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:40:55.522394 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.522362    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zsclz/must-gather-m4xkg"]
Oct 30 19:40:55.522894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.522668    4511 cpu_manager.go:414] "containerMap: removing container" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="storage-initializer"
Oct 30 19:40:55.522894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.522680    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="storage-initializer"
Oct 30 19:40:55.522894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.522695    4511 cpu_manager.go:414] "containerMap: removing container" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kserve-container"
Oct 30 19:40:55.522894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.522702    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kserve-container"
Oct 30 19:40:55.522894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.522709    4511 cpu_manager.go:414] "containerMap: removing container" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kube-rbac-proxy"
Oct 30 19:40:55.522894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.522713    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kube-rbac-proxy"
Oct 30 19:40:55.522894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.522721    4511 cpu_manager.go:414] "containerMap: removing container" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kube-rbac-proxy"
Oct 30 19:40:55.522894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.522726    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kube-rbac-proxy"
Oct 30 19:40:55.522894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.522738    4511 cpu_manager.go:414] "containerMap: removing container" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kserve-container"
Oct 30 19:40:55.522894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.522744    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kserve-container"
Oct 30 19:40:55.522894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.522753    4511 cpu_manager.go:414] "containerMap: removing container" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="storage-initializer"
Oct 30 19:40:55.522894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.522758    4511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="storage-initializer"
Oct 30 19:40:55.522894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.522803    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kserve-container"
Oct 30 19:40:55.522894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.522811    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe9ce467-d2be-48d0-b134-5f6aa7a8259a" containerName="kube-rbac-proxy"
Oct 30 19:40:55.522894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.522817    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kube-rbac-proxy"
Oct 30 19:40:55.522894 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.522825    4511 memory_manager.go:365] "RemoveStaleState removing state" podUID="216ed3b4-58db-4a73-bd92-326baa7b9aff" containerName="kserve-container"
Oct 30 19:40:55.525900 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.525877    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zsclz/must-gather-m4xkg"
Oct 30 19:40:55.576052 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.576029    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zsclz\"/\"kube-root-ca.crt\""
Oct 30 19:40:55.576197 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.576171    4511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zsclz\"/\"openshift-service-ca.crt\""
Oct 30 19:40:55.623354 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.623329    4511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zsclz\"/\"default-dockercfg-cwfbv\""
Oct 30 19:40:55.624555 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.624537    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zsclz/must-gather-m4xkg"]
Oct 30 19:40:55.661991 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.661957    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwzvq\" (UniqueName: \"kubernetes.io/projected/1b221293-8001-4773-a76d-5e84b9b58719-kube-api-access-gwzvq\") pod \"must-gather-m4xkg\" (UID: \"1b221293-8001-4773-a76d-5e84b9b58719\") " pod="openshift-must-gather-zsclz/must-gather-m4xkg"
Oct 30 19:40:55.661991 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.661990    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1b221293-8001-4773-a76d-5e84b9b58719-must-gather-output\") pod \"must-gather-m4xkg\" (UID: \"1b221293-8001-4773-a76d-5e84b9b58719\") " pod="openshift-must-gather-zsclz/must-gather-m4xkg"
Oct 30 19:40:55.762825 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.762799    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwzvq\" (UniqueName: \"kubernetes.io/projected/1b221293-8001-4773-a76d-5e84b9b58719-kube-api-access-gwzvq\") pod \"must-gather-m4xkg\" (UID: \"1b221293-8001-4773-a76d-5e84b9b58719\") " pod="openshift-must-gather-zsclz/must-gather-m4xkg"
Oct 30 19:40:55.762935 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.762836    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1b221293-8001-4773-a76d-5e84b9b58719-must-gather-output\") pod \"must-gather-m4xkg\" (UID: \"1b221293-8001-4773-a76d-5e84b9b58719\") " pod="openshift-must-gather-zsclz/must-gather-m4xkg"
Oct 30 19:40:55.763178 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.763160    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1b221293-8001-4773-a76d-5e84b9b58719-must-gather-output\") pod \"must-gather-m4xkg\" (UID: \"1b221293-8001-4773-a76d-5e84b9b58719\") " pod="openshift-must-gather-zsclz/must-gather-m4xkg"
Oct 30 19:40:55.822432 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.822408    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwzvq\" (UniqueName: \"kubernetes.io/projected/1b221293-8001-4773-a76d-5e84b9b58719-kube-api-access-gwzvq\") pod \"must-gather-m4xkg\" (UID: \"1b221293-8001-4773-a76d-5e84b9b58719\") " pod="openshift-must-gather-zsclz/must-gather-m4xkg"
Oct 30 19:40:55.849278 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:55.849255    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zsclz/must-gather-m4xkg"
Oct 30 19:40:56.216676 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:56.216598    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zsclz/must-gather-m4xkg"]
Oct 30 19:40:56.224143 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:40:56.224104    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b221293_8001_4773_a76d_5e84b9b58719.slice/crio-a869791b0e62c0042c4a5108fd9d6370bba7c511a33e7e7e813680341a9f1c8a WatchSource:0}: Error finding container a869791b0e62c0042c4a5108fd9d6370bba7c511a33e7e7e813680341a9f1c8a: Status 404 returned error can't find the container with id a869791b0e62c0042c4a5108fd9d6370bba7c511a33e7e7e813680341a9f1c8a
Oct 30 19:40:56.225808 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:56.225789    4511 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:40:57.213753 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:57.213718    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zsclz/must-gather-m4xkg" event={"ID":"1b221293-8001-4773-a76d-5e84b9b58719","Type":"ContainerStarted","Data":"a869791b0e62c0042c4a5108fd9d6370bba7c511a33e7e7e813680341a9f1c8a"}
Oct 30 19:40:58.218856 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:58.218824    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zsclz/must-gather-m4xkg" event={"ID":"1b221293-8001-4773-a76d-5e84b9b58719","Type":"ContainerStarted","Data":"d566302179dec30f80d3d4efd279e3ccd46b2c383587aaebff43f0f0ee4a2424"}
Oct 30 19:40:58.218856 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:58.218865    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zsclz/must-gather-m4xkg" event={"ID":"1b221293-8001-4773-a76d-5e84b9b58719","Type":"ContainerStarted","Data":"19bc37abb73211644d3f797ec971cdcc02a64feb54317333687bf99608be2f42"}
Oct 30 19:40:58.335363 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:40:58.335310    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zsclz/must-gather-m4xkg" podStartSLOduration=1.9531149939999999 podStartE2EDuration="3.335293736s" totalImagesPullingTime="1.382178742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:58.334713622 +0000 UTC m=+3752.852398523" watchObservedRunningTime="2026-10-30 19:40:58.335293736 +0000 UTC m=+3752.852978635"
Oct 30 19:41:05.137056 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:05.137027    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bwbj7_d197c973-f7cd-4dff-b621-71fc64897eeb/global-pull-secret-syncer/0.log"
Oct 30 19:41:08.195189 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:08.195112    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dvpdt_5ff8c95a-a5f2-41d1-91f3-ff288ebf2e5b/konnectivity-agent/0.log"
Oct 30 19:41:09.962750 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:09.962720    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-66.us-west-2.compute.internal_80e81f6072b8791c05f541935e51a65e/haproxy/0.log"
Oct 30 19:41:10.158769 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:41:10.158731    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:41:18.068396 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.068365    4511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"]
Oct 30 19:41:18.073408 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.073388    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:18.121034 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.121002    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46c5b611-f35e-4a30-8962-333e2bd997b2-lib-modules\") pod \"perf-node-gather-daemonset-zq2np\" (UID: \"46c5b611-f35e-4a30-8962-333e2bd997b2\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:18.121214 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.121043    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46c5b611-f35e-4a30-8962-333e2bd997b2-podres\") pod \"perf-node-gather-daemonset-zq2np\" (UID: \"46c5b611-f35e-4a30-8962-333e2bd997b2\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:18.121214 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.121065    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46c5b611-f35e-4a30-8962-333e2bd997b2-proc\") pod \"perf-node-gather-daemonset-zq2np\" (UID: \"46c5b611-f35e-4a30-8962-333e2bd997b2\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:18.121347 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.121276    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46c5b611-f35e-4a30-8962-333e2bd997b2-sys\") pod \"perf-node-gather-daemonset-zq2np\" (UID: \"46c5b611-f35e-4a30-8962-333e2bd997b2\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:18.121408 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.121385    4511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llrcn\" (UniqueName: \"kubernetes.io/projected/46c5b611-f35e-4a30-8962-333e2bd997b2-kube-api-access-llrcn\") pod \"perf-node-gather-daemonset-zq2np\" (UID: \"46c5b611-f35e-4a30-8962-333e2bd997b2\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:18.163880 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.163854    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"]
Oct 30 19:41:18.222621 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.222593    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llrcn\" (UniqueName: \"kubernetes.io/projected/46c5b611-f35e-4a30-8962-333e2bd997b2-kube-api-access-llrcn\") pod \"perf-node-gather-daemonset-zq2np\" (UID: \"46c5b611-f35e-4a30-8962-333e2bd997b2\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:18.222799 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.222634    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46c5b611-f35e-4a30-8962-333e2bd997b2-lib-modules\") pod \"perf-node-gather-daemonset-zq2np\" (UID: \"46c5b611-f35e-4a30-8962-333e2bd997b2\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:18.222799 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.222658    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46c5b611-f35e-4a30-8962-333e2bd997b2-podres\") pod \"perf-node-gather-daemonset-zq2np\" (UID: \"46c5b611-f35e-4a30-8962-333e2bd997b2\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:18.222799 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.222699    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46c5b611-f35e-4a30-8962-333e2bd997b2-proc\") pod \"perf-node-gather-daemonset-zq2np\" (UID: \"46c5b611-f35e-4a30-8962-333e2bd997b2\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:18.222799 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.222746    4511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46c5b611-f35e-4a30-8962-333e2bd997b2-sys\") pod \"perf-node-gather-daemonset-zq2np\" (UID: \"46c5b611-f35e-4a30-8962-333e2bd997b2\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:18.222799 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.222802    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46c5b611-f35e-4a30-8962-333e2bd997b2-lib-modules\") pod \"perf-node-gather-daemonset-zq2np\" (UID: \"46c5b611-f35e-4a30-8962-333e2bd997b2\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:18.223043 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.222818    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46c5b611-f35e-4a30-8962-333e2bd997b2-proc\") pod \"perf-node-gather-daemonset-zq2np\" (UID: \"46c5b611-f35e-4a30-8962-333e2bd997b2\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:18.223043 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.222824    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46c5b611-f35e-4a30-8962-333e2bd997b2-podres\") pod \"perf-node-gather-daemonset-zq2np\" (UID: \"46c5b611-f35e-4a30-8962-333e2bd997b2\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:18.223043 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.222872    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46c5b611-f35e-4a30-8962-333e2bd997b2-sys\") pod \"perf-node-gather-daemonset-zq2np\" (UID: \"46c5b611-f35e-4a30-8962-333e2bd997b2\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:18.282606 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.282567    4511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llrcn\" (UniqueName: \"kubernetes.io/projected/46c5b611-f35e-4a30-8962-333e2bd997b2-kube-api-access-llrcn\") pod \"perf-node-gather-daemonset-zq2np\" (UID: \"46c5b611-f35e-4a30-8962-333e2bd997b2\") " pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:18.385743 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.385668    4511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:18.754700 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:18.754624    4511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"]
Oct 30 19:41:18.762450 ip-10-0-135-66 kubenswrapper[4511]: W1030 19:41:18.762414    4511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod46c5b611_f35e_4a30_8962_333e2bd997b2.slice/crio-621fab93fcbd621b173adaddba4d189aa1f325bae97044a78918dea729936538 WatchSource:0}: Error finding container 621fab93fcbd621b173adaddba4d189aa1f325bae97044a78918dea729936538: Status 404 returned error can't find the container with id 621fab93fcbd621b173adaddba4d189aa1f325bae97044a78918dea729936538
Oct 30 19:41:19.316185 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:19.316146    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np" event={"ID":"46c5b611-f35e-4a30-8962-333e2bd997b2","Type":"ContainerStarted","Data":"009a71ed9941946d7e767b1ac7184c06c89210fe979d0dffd8cef923e73cb219"}
Oct 30 19:41:19.316687 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:19.316209    4511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np" event={"ID":"46c5b611-f35e-4a30-8962-333e2bd997b2","Type":"ContainerStarted","Data":"621fab93fcbd621b173adaddba4d189aa1f325bae97044a78918dea729936538"}
Oct 30 19:41:19.316944 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:19.316920    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:19.429137 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:19.429076    4511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np" podStartSLOduration=1.429062853 podStartE2EDuration="1.429062853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:19.428260879 +0000 UTC m=+3773.945945779" watchObservedRunningTime="2026-10-30 19:41:19.429062853 +0000 UTC m=+3773.946747754"
Oct 30 19:41:19.534571 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:19.534528    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-99qt8_58ff6b78-bcbb-4759-bd03-3a0537510e42/monitoring-plugin/0.log"
Oct 30 19:41:22.424462 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:22.424419    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zpscf_f83aee63-bc05-4a59-8ca1-16abf2d6b7fe/node-exporter/0.log"
Oct 30 19:41:22.833451 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:22.833427    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zpscf_f83aee63-bc05-4a59-8ca1-16abf2d6b7fe/kube-rbac-proxy/0.log"
Oct 30 19:41:23.244876 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:23.244803    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zpscf_f83aee63-bc05-4a59-8ca1-16abf2d6b7fe/init-textfile/0.log"
Oct 30 19:41:26.333215 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:26.333185    4511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zsclz/perf-node-gather-daemonset-zq2np"
Oct 30 19:41:28.634180 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:28.634153    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-pvfhh_cd5a8461-de4f-4e73-bf84-96577e299b46/prometheus-operator-admission-webhook/0.log"
Oct 30 19:41:40.026218 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:40.026189    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-766767df9c-vjsnr_cc31efc4-3e87-4b47-8c76-b71118537f72/console/0.log"
Oct 30 19:41:40.156662 ip-10-0-135-66 kubenswrapper[4511]: E1030 19:41:40.156626    4511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jhgk" containerName="extractor"
Oct 30 19:41:40.493881 ip-10-0-135-66 kubenswrapper[4511]: I1030 19:41:40.493854    4511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-zr2j7_4e09ad47-0f10-4b7d-a229-13390069f458/download-server/0.log"