Oct 30 22:07:39.655650 ip-10-0-133-218 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 22:07:40.064571 ip-10-0-133-218 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:07:40.064571 ip-10-0-133-218 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 22:07:40.064571 ip-10-0-133-218 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:07:40.064571 ip-10-0-133-218 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:07:40.070881 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070803    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:07:40.070881 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070881    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:07:40.070881 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070885    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070888    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070890    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070893    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070896    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070898    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070901    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070904    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070907    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070910    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070916    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070919    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070922    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070924    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070927    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070930    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070932    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070935    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070938    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070940    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:07:40.070978 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070943    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070945    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070948    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070950    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070953    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070955    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070958    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070960    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070963    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070965    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070968    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070970    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070973    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070975    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070978    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070981    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070984    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070987    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070989    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070992    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:07:40.071459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070995    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.070998    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071002    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071006    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071009    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071011    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071015    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071018    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071020    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071023    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071026    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071028    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071031    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071033    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071035    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071039    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071043    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071046    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:07:40.071915 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071049    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071052    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071054    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071056    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071059    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071062    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071065    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071067    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071070    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071073    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071076    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071078    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071081    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071085    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071087    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071090    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071092    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071095    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071098    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:07:40.072326 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071101    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071103    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071105    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071108    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071110    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071113    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071115    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071118    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071120    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071123    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071125    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071128    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071131    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071133    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071136    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071138    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071141    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071144    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071146    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:07:40.072796 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071149    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071151    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071155    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071158    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071160    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071162    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071165    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071167    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071983    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071990    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071994    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.071997    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072000    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072003    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072006    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072008    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072011    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072013    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072016    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072019    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:07:40.073232 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072021    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072024    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072026    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072031    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072033    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072036    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072039    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072041    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072045    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072048    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072051    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072053    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072056    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072058    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072061    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072063    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072066    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072069    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072071    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072074    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:07:40.073724 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072076    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072078    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072081    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072083    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072086    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072088    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072091    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072094    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072098    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072102    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072105    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072108    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072111    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072114    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072116    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072120    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072123    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072125    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072128    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:07:40.074211 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072130    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072133    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072135    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072138    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072140    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072143    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072145    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072148    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072150    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072152    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072155    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072157    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072160    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072162    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072165    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072167    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072169    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072172    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072175    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072178    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:07:40.074664 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072181    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072183    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072186    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072188    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072191    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072194    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072196    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072198    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072202    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072206    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072208    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072212    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072215    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072218    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072220    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072223    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072226    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072228    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072230    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:07:40.075133 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072233    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072236    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072238    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072241    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072243    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072246    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072248    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072251    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072254    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072257    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072259    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072262    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072265    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072268    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072272    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072275    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072765    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072771    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072774    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072777    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:07:40.075580 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072779    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072782    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072785    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072788    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072791    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072793    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072796    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072798    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072801    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072803    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072806    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072809    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072811    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072814    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072818    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072821    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072825    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072828    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072830    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:07:40.076064 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072833    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072835    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072838    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072841    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072844    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072846    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072849    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072852    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072855    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072858    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072860    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072863    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072865    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072867    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072870    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072872    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072875    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072877    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072880    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:07:40.076521 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072882    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072885    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072887    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072889    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072892    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072894    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072896    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072899    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072901    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072904    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072906    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072910    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072914    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072917    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072920    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072922    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072925    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072928    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072930    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:07:40.076963 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072933    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072935    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072938    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072941    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072943    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072946    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072948    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072951    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072954    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072956    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072959    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072961    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072964    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072966    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072969    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072971    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072974    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072976    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072979    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:07:40.077461 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072982    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072985    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072987    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072990    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072992    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072995    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072997    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.072999    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073002    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073004    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073007    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073009    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073012    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073014    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073016    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073019    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073022    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073024    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073027    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073029    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:07:40.077932 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073031    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073034    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073036    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073038    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073041    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.073043    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073840    2572 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073849    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073855    2572 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073860    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073864    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073867    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073871    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073875    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073878    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073881    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073885    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073888    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073891    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073894    2572 flags.go:64] FLAG: --cgroup-root=""
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073896    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073899    2572 flags.go:64] FLAG: --client-ca-file=""
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073902    2572 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073905    2572 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 22:07:40.078409 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073913    2572 flags.go:64] FLAG: --cluster-domain=""
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073916    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073919    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073922    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073925    2572 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073929    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073932    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073935    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073938    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073941    2572 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073944    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073947    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073950    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073953    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073957    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073960    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073963    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073966    2572 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073969    2572 flags.go:64] FLAG: --enable-server="true"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073973    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073976    2572 flags.go:64] FLAG: --event-burst="100"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073979    2572 flags.go:64] FLAG: --event-qps="50"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073982    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073985    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 22:07:40.078986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073990    2572 flags.go:64] FLAG: --eviction-hard=""
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073994    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.073997    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074000    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074003    2572 flags.go:64] FLAG: --eviction-soft=""
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074006    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074009    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074012    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074017    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074020    2572 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074023    2572 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074025    2572 flags.go:64] FLAG: --feature-gates=""
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074030    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074033    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074036    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074040    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074043    2572 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074046    2572 flags.go:64] FLAG: --help="false"
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074048    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074052    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074055    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074058    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074061    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 22:07:40.079567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074064    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074067    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074070    2572 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074073    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074076    2572 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074079    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074082    2572 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074085    2572 flags.go:64] FLAG: --kube-reserved=""
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074088    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074092    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074095    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074098    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074101    2572 flags.go:64] FLAG: --lock-file=""
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074104    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074107    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074110    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074115    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074119    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074122    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074124    2572 flags.go:64] FLAG: --logging-format="text"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074127    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074131    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074133    2572 flags.go:64] FLAG: --manifest-url=""
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074136    2572 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074141    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 22:07:40.080107 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074145    2572 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074149    2572 flags.go:64] FLAG: --max-pods="110"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074152    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074155    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074158    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074161    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074164    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074167    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074170    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074177    2572 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074180    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074183    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074186    2572 flags.go:64] FLAG: --pod-cidr=""
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074189    2572 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074191    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074195    2572 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074199    2572 flags.go:64] FLAG: --port="10250"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074202    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074205    2572 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0b3d36b757523b2ee"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074208    2572 flags.go:64] FLAG: --qos-reserved=""
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074211    2572 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074214    2572 flags.go:64] FLAG: --register-node="true"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074217    2572 flags.go:64] FLAG: --register-with-taints=""
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074220    2572 flags.go:64] FLAG: --registry-burst="10"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074225    2572 flags.go:64] FLAG: --registry-qps="5"
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074228    2572 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 22:07:40.080979 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074231    2572 flags.go:64] FLAG: --reserved-memory=""
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074234    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074238    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074241    2572 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074243    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074246    2572 flags.go:64] FLAG: --runonce="false"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074250    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074253    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074256    2572 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074259    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074262    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074265    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074268    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074271    2572 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074273    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074276    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074279    2572 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074282    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074285    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074288    2572 flags.go:64] FLAG: --system-cgroups=""
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074291    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074296    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074300    2572 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074303    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074307    2572 flags.go:64] FLAG: --tls-min-version=""
Oct 30 22:07:40.081992 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074310    2572 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074313    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074315    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074318    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074321    2572 flags.go:64] FLAG: --v="2"
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074340    2572 flags.go:64] FLAG: --version="false"
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074344    2572 flags.go:64] FLAG: --vmodule=""
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074348    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074352    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074486    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074490    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074492    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074495    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074497    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074500    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074503    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074506    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074508    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074510    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074513    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074515    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:07:40.082645 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074518    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074520    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074523    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074525    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074528    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074530    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074533    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074536    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074540    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074542    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074545    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074548    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074550    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074552    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074555    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074557    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074561    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074564    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074567    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074569    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:07:40.083141 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074571    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074574    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074576    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074579    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074581    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074585    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074589    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074593    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074596    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074600    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074604    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074606    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074609    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074611    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074613    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074616    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074618    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074621    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074623    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:07:40.083638 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074625    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074629    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074632    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074635    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074637    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074640    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074642    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074645    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074648    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074651    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074654    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074656    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074659    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074661    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074664    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074666    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074669    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074672    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074674    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:07:40.084079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074677    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074680    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074682    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074685    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074687    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074690    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074693    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074695    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074697    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074700    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074702    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074705    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074707    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074710    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074713    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074716    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074718    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074721    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074723    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:07:40.084540 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074725    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:07:40.084983 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074728    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:07:40.084983 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074731    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:07:40.084983 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074734    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:07:40.084983 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074737    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:07:40.084983 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074739    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:07:40.084983 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074742    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:07:40.084983 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074744    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:07:40.084983 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074746    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:07:40.084983 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074749    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:07:40.084983 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074751    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:07:40.084983 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074754    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:07:40.084983 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074756    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:07:40.084983 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074759    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:07:40.084983 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074761    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:07:40.084983 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074764    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:07:40.084983 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074767    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:07:40.085359 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.074792    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:07:40.085359 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074830    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:07:40.085359 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074835    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:07:40.085359 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074837    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:07:40.085359 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074840    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:07:40.085359 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074843    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:07:40.085359 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074845    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:07:40.085359 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074848    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:07:40.085359 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074850    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:07:40.085359 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074852    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:07:40.085359 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074855    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:07:40.085359 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074859    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:07:40.085359 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074862    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:07:40.085359 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074864    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:07:40.085359 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074867    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:07:40.085359 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074869    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074872    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074874    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074878    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074880    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074883    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074886    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074890    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074893    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074896    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074900    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074903    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074906    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074909    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074911    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074914    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074917    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074919    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074922    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:07:40.085749 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074924    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074927    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074929    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074932    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074934    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074937    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074939    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074941    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074944    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074948    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074951    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074953    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074955    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074958    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074961    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074963    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074968    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074970    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074973    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:07:40.086196 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074975    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074978    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074981    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074983    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074986    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074988    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074990    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074993    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074996    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.074998    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075001    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075004    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075006    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075009    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075011    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075014    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075016    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075018    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075021    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075023    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:07:40.086655 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075025    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075029    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075032    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075035    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075038    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075040    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075042    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075045    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075048    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075052    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075054    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075057    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075059    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075062    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075064    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075066    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075069    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075071    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075074    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075076    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:07:40.087112 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075078    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075081    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075083    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075086    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075088    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075091    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075093    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075096    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075098    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075101    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075103    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075105    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075108    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.075110    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.083607    2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.083625    2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.084161    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.084168    2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084231    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:07:40.087626 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084237    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084240    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084243    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084246    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084249    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084251    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084254    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084256    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084259    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084262    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084265    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084267    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084269    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084272    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084275    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084277    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084280    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084283    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084286    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084288    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:07:40.088103 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084291    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084293    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084302    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084305    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084308    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084310    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084313    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084315    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084318    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084321    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084324    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084328    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084352    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084356    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084360    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084363    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084366    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084368    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:07:40.088602 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084371    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084373    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084376    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084379    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084382    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084384    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084387    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084390    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084393    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084395    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084398    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084400    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084403    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084405    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084408    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084410    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084414    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084416    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084419    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084421    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:07:40.089016 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084424    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084426    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084429    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084432    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084434    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084437    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084439    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084442    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084444    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084446    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084449    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084451    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084454    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084456    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084459    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084461    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084464    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084466    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084469    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084471    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:07:40.089644 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084474    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084477    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084480    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084482    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084484    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084487    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084489    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084492    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084494    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084497    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084499    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084502    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084504    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084507    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084510    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084512    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084515    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084517    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084519    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084522    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:07:40.090131 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084524    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:07:40.090622 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084527    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:07:40.090622 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084529    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:07:40.090622 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084531    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:07:40.090622 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084534    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:07:40.090622 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084536    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:07:40.090622 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084538    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:07:40.090622 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.084566    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:07:40.090622 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084708    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:07:40.090622 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084713    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:07:40.090622 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084716    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:07:40.090622 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084719    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:07:40.090622 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084722    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:07:40.090622 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084725    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:07:40.090622 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084727    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:07:40.090622 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084730    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:07:40.090622 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084732    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084735    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084737    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084740    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084742    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084745    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084747    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084750    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084752    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084755    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084758    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084760    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084763    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084765    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084768    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084770    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084773    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084775    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084778    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084780    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:07:40.090988 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084783    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084785    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084787    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084790    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084792    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084795    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084797    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084800    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084802    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084805    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084808    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084810    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084813    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084815    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084818    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084820    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084823    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084825    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084828    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084830    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:07:40.091506 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084833    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084836    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084838    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084841    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084843    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084846    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084848    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084851    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084853    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084855    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084858    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084860    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084863    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084866    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084868    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084871    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084873    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084876    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084878    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084880    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:07:40.091967 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084883    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084885    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084888    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084891    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084895    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084898    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084901    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084904    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084906    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084910    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084914    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084917    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084920    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084923    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084925    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084928    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084930    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084932    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:07:40.092460 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084935    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084937    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084940    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084942    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084945    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084947    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084950    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084952    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084955    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084957    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084960    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084962    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084964    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084967    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084970    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084972    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084975    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084977    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084980    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:07:40.092862 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.084983    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:07:40.093292 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.085006    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:07:40.093292 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085038    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:07:40.093292 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085042    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:07:40.093292 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085045    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:07:40.093292 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085048    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:07:40.093292 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085051    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:07:40.093292 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085053    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:07:40.093292 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085056    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:07:40.093292 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085059    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:07:40.093292 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085062    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:07:40.093292 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085066    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:07:40.093292 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085069    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:07:40.093292 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085072    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:07:40.093292 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085074    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:07:40.093292 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085080    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085085    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085088    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085091    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085093    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085096    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085098    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085101    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085103    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085106    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085109    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085111    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085114    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085116    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085118    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085121    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085124    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085127    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085130    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:07:40.093742 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085132    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085135    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085137    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085140    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085142    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085145    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085147    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085150    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085153    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085155    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085159    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085162    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085164    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085169    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085171    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085174    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085176    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085179    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085181    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085183    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:07:40.094177 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085186    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085188    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085191    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085193    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085196    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085198    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085200    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085204    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085207    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085209    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085212    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085215    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085217    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085220    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085222    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085225    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085227    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085229    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085232    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085235    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:07:40.094677 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085237    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085240    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085242    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085245    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085247    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085251    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085253    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085256    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085258    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085261    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085264    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085266    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085269    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085271    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085274    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085277    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085280    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085283    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085285    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085288    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:07:40.095124 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085290    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085293    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085295    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085298    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085304    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085307    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085309    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085312    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085314    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085317    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085319    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085322    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085324    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.085326    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.085404    2572 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.087435    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.088392    2572 server.go:1014] "Starting client certificate rotation"
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.088484    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.088520    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:07:40.095600 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.090090    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:07:40.096031 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.094798    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:07:40.103526 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.103508    2572 log.go:25] "Validated CRI v1 runtime API"
Oct 30 22:07:40.106975 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.106960    2572 log.go:25] "Validated CRI v1 image API"
Oct 30 22:07:40.107848 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.107826    2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 22:07:40.112077 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.112051    2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8e076779-6cd5-400d-b4ca-994c866b4b29:/dev/nvme0n1p4 e42f82d1-72e2-4427-8c5a-3cd580b53f1b:/dev/nvme0n1p3]
Oct 30 22:07:40.112128 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.112077    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 22:07:40.118191 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.118069    2572 manager.go:217] Machine: {Timestamp:2026-10-30 22:07:40.117605666 +0000 UTC m=+0.326238330 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100705 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f0d7b59cb9095c25be4cfad65344c SystemUUID:ec2f0d7b-59cb-9095-c25b-e4cfad65344c BootID:63d5c887-70ed-4365-9d07-f49adfb96172 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0a:42:03:3c:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0a:42:03:3c:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:92:69:c2:2c:f3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 22:07:40.118191 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.118177    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 22:07:40.118355 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.118253    2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 22:07:40.119352 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.119310    2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 22:07:40.119488 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.119354    2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-218.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 22:07:40.119529 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.119490    2572 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 22:07:40.119529 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.119499    2572 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 22:07:40.119529 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.119523    2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:07:40.119607 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.119536    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:07:40.120552 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.120541    2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 22:07:40.124322 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.124310    2572 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 22:07:40.124464 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.124454    2572 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 22:07:40.125013 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.125003    2572 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 22:07:40.125056 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.125019    2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 22:07:40.125056 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.125040    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 22:07:40.125056 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.125053    2572 kubelet.go:405] "Adding apiserver pod source"
Oct 30 22:07:40.125190 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.125061    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 22:07:40.125462 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.125447    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:07:40.127385 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.127368    2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 22:07:40.132673 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.132656    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:07:40.133245 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.133231    2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 22:07:40.133312 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.133273    2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 22:07:40.133499 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.133487    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 22:07:40.133572 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.133507    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 22:07:40.133572 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.133523    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 22:07:40.133572 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.133534    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 22:07:40.133572 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.133542    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 22:07:40.133572 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.133551    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 22:07:40.133862 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.133852    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 22:07:40.133911 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.133866    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 22:07:40.133911 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.133878    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 22:07:40.133911 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.133888    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 22:07:40.133911 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.133902    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 22:07:40.134082 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.133917    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 22:07:40.134998 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.134987    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 22:07:40.135062 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.135002    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 22:07:40.138783 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.138768    2572 server.go:1263] "Started kubelet"
Oct 30 22:07:40.138869 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.138821    2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 22:07:40.139503 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.139451    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 22:07:40.139575 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.139528    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 22:07:40.140194 ip-10-0-133-218 systemd[1]: Started Kubernetes Kubelet.
Oct 30 22:07:40.140906 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.140811    2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 22:07:40.142227 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.142210    2572 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 22:07:40.144940 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:40.144916    2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 22:07:40.147217 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.147205    2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 22:07:40.147307 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.147287    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 22:07:40.147388 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:40.147347    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-218.us-west-1.compute.internal\" not found"
Oct 30 22:07:40.147866 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.147848    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 22:07:40.148158 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.147853    2572 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 22:07:40.148272 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.148164    2572 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 22:07:40.148475 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.148303    2572 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 22:07:40.148475 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.148310    2572 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 22:07:40.148688 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.148670    2572 factory.go:55] Registering systemd factory
Oct 30 22:07:40.148743 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.148694    2572 factory.go:223] Registration of the systemd container factory successfully
Oct 30 22:07:40.148998 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.148981    2572 factory.go:153] Registering CRI-O factory
Oct 30 22:07:40.149065 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.149017    2572 factory.go:223] Registration of the crio container factory successfully
Oct 30 22:07:40.149123 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.149100    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 22:07:40.149123 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.149119    2572 factory.go:103] Registering Raw factory
Oct 30 22:07:40.149217 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.149176    2572 manager.go:1199] Started watching for new ooms in manager
Oct 30 22:07:40.150171 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.150121    2572 manager.go:319] Starting recovery of all containers
Oct 30 22:07:40.157407 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.157355    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 22:07:40.162385 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:40.162323    2572 helpers.go:246] readString: Failed to read "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service/memory.min": read /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service/memory.min: no such device
Oct 30 22:07:40.163514 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.163496    2572 manager.go:324] Recovery completed
Oct 30 22:07:40.167161 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.167147    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:07:40.169620 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.169597    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:07:40.169693 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.169637    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:07:40.169693 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.169653    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:07:40.170166 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.170148    2572 cpu_manager.go:226] "Starting" policy="none"
Oct 30 22:07:40.170166 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.170165    2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 22:07:40.170283 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.170183    2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 22:07:40.170283 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.170139    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:07:40.170927 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.170913    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:07:40.170986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.170938    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:07:40.170986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.170948    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:07:40.172237 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.172226    2572 policy_none.go:50] "Start"
Oct 30 22:07:40.172280 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.172241    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 22:07:40.172280 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.172252    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 22:07:40.174152 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.174142    2572 policy_none.go:44] "Start"
Oct 30 22:07:40.236737 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.216864    2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 22:07:40.236737 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.219189    2572 manager.go:341] "Starting Device Plugin manager"
Oct 30 22:07:40.236737 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:40.219646    2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 22:07:40.236737 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.219670    2572 server.go:87] "Starting device plugin registration server"
Oct 30 22:07:40.236737 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.219925    2572 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 22:07:40.236737 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.219937    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 22:07:40.236737 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.220103    2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 22:07:40.236737 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.220174    2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 22:07:40.236737 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.220179    2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 22:07:40.236737 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:40.220645    2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 22:07:40.236737 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:40.220684    2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-218.us-west-1.compute.internal\" not found"
Oct 30 22:07:40.236737 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.228147    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 22:07:40.236737 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.228167    2572 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 22:07:40.236737 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.228183    2572 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 22:07:40.236737 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:40.228212    2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 22:07:40.321041 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.320863    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:07:40.321943 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.321927    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:07:40.322036 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.321967    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:07:40.322036 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.321982    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:07:40.322036 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.322012    2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.328524 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.328506    2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"]
Oct 30 22:07:40.328608 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.328565    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:07:40.329459 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.329442    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:07:40.329531 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.329469    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:07:40.329531 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.329480    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:07:40.331697 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.331685    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:07:40.331857 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.331840    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.331893 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.331873    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:07:40.332447 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.332430    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:07:40.332511 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.332463    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:07:40.332511 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.332474    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:07:40.332511 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.332438    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:07:40.332624 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.332533    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:07:40.332624 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.332544    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:07:40.333742 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:40.333598    2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-218.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 22:07:40.333742 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.333632    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-218.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 22:07:40.336401 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:40.333689    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-218.us-west-1.compute.internal.18e36e08ab706333  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-218.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-218.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 22:07:40.138652467 +0000 UTC m=+0.347285130,LastTimestamp:2026-10-30 22:07:40.138652467 +0000 UTC m=+0.347285130,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-218.us-west-1.compute.internal,}"
Oct 30 22:07:40.336519 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.336425    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.336519 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.336471    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:07:40.338532 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.338511    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:07:40.338627 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.338551    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:07:40.338627 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.338565    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:07:40.343873 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:40.343849    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 22:07:40.343984 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:40.343882    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 30 22:07:40.349741 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.349723    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aaa975e9f9cee63edf4d69aef6d66c72-config\") pod \"kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"aaa975e9f9cee63edf4d69aef6d66c72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.349810 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.349750    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/782d33a65ccf759ca199fdbbd0e7fdf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"782d33a65ccf759ca199fdbbd0e7fdf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.349810 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.349765    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/782d33a65ccf759ca199fdbbd0e7fdf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"782d33a65ccf759ca199fdbbd0e7fdf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.369286 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:40.369261    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-218.us-west-1.compute.internal\" not found" node="ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.375976 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:40.375958    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-218.us-west-1.compute.internal\" not found" node="ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.376069 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.376053    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 22:07:40.376129 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.376113    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4wj7s"
Oct 30 22:07:40.450091 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.450050    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aaa975e9f9cee63edf4d69aef6d66c72-config\") pod \"kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"aaa975e9f9cee63edf4d69aef6d66c72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.450264 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.450096    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/782d33a65ccf759ca199fdbbd0e7fdf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"782d33a65ccf759ca199fdbbd0e7fdf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.450264 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.450123    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/782d33a65ccf759ca199fdbbd0e7fdf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"782d33a65ccf759ca199fdbbd0e7fdf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.450264 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.450140    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aaa975e9f9cee63edf4d69aef6d66c72-config\") pod \"kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"aaa975e9f9cee63edf4d69aef6d66c72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.450264 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.450166    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/782d33a65ccf759ca199fdbbd0e7fdf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"782d33a65ccf759ca199fdbbd0e7fdf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.450264 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.450223    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/782d33a65ccf759ca199fdbbd0e7fdf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"782d33a65ccf759ca199fdbbd0e7fdf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.525692 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.525663    2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:07:40.573795 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.573743    2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:07:40.574477 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.574462    2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.574526 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:40.574485    2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-218.us-west-1.compute.internal\": node \"ip-10-0-133-218.us-west-1.compute.internal\" not found"
Oct 30 22:07:40.647601 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.647576    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.674004 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.673984    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.679893 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.679230    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.798675 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.798650    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:07:40.799441 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.799429    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:07:40.899765 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:40.899694    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:07:41.088758 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.088728    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 22:07:41.089204 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.088933    2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 22:07:41.089204 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:41.088966    2572 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a6f4d41d4389f4fb28595b18760ef362-f326b7236f2e3088.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-218.us-west-1.compute.internal.18e36e08ad494738\": read tcp 10.0.133.218:38212->3.12.159.144:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-218.us-west-1.compute.internal.18e36e08ad494738  default   4755 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-218.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-218.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-218.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 22:07:40 +0000 UTC,LastTimestamp:2026-10-30 22:07:40.321973839 +0000 UTC m=+0.530606511,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-218.us-west-1.compute.internal,}"
Oct 30 22:07:41.125864 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.125837    2572 apiserver.go:52] "Watching apiserver"
Oct 30 22:07:41.148396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.148371    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 22:07:41.383037 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.382945    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 22:02:40 +0000 UTC" deadline="2028-04-01 18:08:53.174058769 +0000 UTC"
Oct 30 22:07:41.383037 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.382988    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12452h1m11.791074071s"
Oct 30 22:07:41.394776 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.394747    2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 22:07:41.395029 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.395012    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl","openshift-cluster-node-tuning-operator/tuned-np6dm","openshift-network-operator/iptables-alerter-c5whj","kube-system/konnectivity-agent-fgkn6","kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal","openshift-dns/node-resolver-s2528","openshift-image-registry/node-ca-bc87g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal","openshift-multus/multus-2ptq8","openshift-ovn-kubernetes/ovnkube-node-z2zrp","kube-system/global-pull-secret-syncer-xtgwp","openshift-multus/multus-additional-cni-plugins-cs9rl","openshift-multus/network-metrics-daemon-g5pws","openshift-network-diagnostics/network-check-target-7h6kb"]
Oct 30 22:07:41.397639 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.397625    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.400429 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.400399    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:07:41.402518 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.402502    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:07:41.404669 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.404656    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.406655 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.406643    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:41.406721 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:41.406703    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g5pws" podUID="3788135a-d420-4738-a292-845c50e486a7"
Oct 30 22:07:41.408600 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.408586    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s2528"
Oct 30 22:07:41.410688 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.410675    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.412972 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.412955    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.415299 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.415285    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.417533 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.417508    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:41.417602 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:41.417577    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtgwp" podUID="93f981a6-4af6-4405-aec6-89e539c42c67"
Oct 30 22:07:41.419786 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.419768    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fgkn6"
Oct 30 22:07:41.422040 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.422023    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:07:41.422093 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:41.422079    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h6kb" podUID="7c1575a8-9288-4779-aab6-e5409b097e9f"
Oct 30 22:07:41.448814 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.448794    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 22:07:41.450977 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.450959    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 22:07:41.451146 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.451026    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 22:07:41.451146 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.451029    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 22:07:41.451289 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.451184    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 22:07:41.451289 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.451233    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 22:07:41.451289 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.451276    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 22:07:41.451564 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.451551    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lds9w\""
Oct 30 22:07:41.452773 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.452755    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 22:07:41.452773 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.452764    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 22:07:41.452886 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.452829    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lmcdt\""
Oct 30 22:07:41.452921 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.452893    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 22:07:41.454749 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.454733    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:07:41.454880 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.454868    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 22:07:41.456036 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456022    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-cni-binary-copy\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.456085 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456044    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-cni-binary-copy\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.456085 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456060    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-cni-multus\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.456085 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456074    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-cni-netd\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.456200 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456091    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.456200 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456129    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2q5s\" (UniqueName: \"kubernetes.io/projected/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-kube-api-access-j2q5s\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.456200 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456159    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xllgn\" (UniqueName: \"kubernetes.io/projected/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-kube-api-access-xllgn\") pod \"node-resolver-s2528\" (UID: \"f92ebb16-2530-4f59-98d6-c3c2695ad9ba\") " pod="openshift-dns/node-resolver-s2528"
Oct 30 22:07:41.456200 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456190    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41dcf151-1d72-4978-b0aa-2046f213cb7e-serviceca\") pod \"node-ca-bc87g\" (UID: \"41dcf151-1d72-4978-b0aa-2046f213cb7e\") " pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:07:41.456396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456243    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-systemd\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.456396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456274    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-hosts-file\") pod \"node-resolver-s2528\" (UID: \"f92ebb16-2530-4f59-98d6-c3c2695ad9ba\") " pod="openshift-dns/node-resolver-s2528"
Oct 30 22:07:41.456396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456300    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-kubelet\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.456396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456327    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-daemon-config\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.456396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456366    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-multus-certs\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.456574 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456407    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-sys-fs\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.456574 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456449    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nskz5\" (UniqueName: \"kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5\") pod \"network-check-target-7h6kb\" (UID: \"7c1575a8-9288-4779-aab6-e5409b097e9f\") " pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:07:41.456574 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456484    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-node-log\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.456574 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456508    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-cni-bin\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.456574 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456525    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/79823d82-b501-47c9-acae-667e192d721b-ovn-node-metrics-cert\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.456574 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456551    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkbv8\" (UniqueName: \"kubernetes.io/projected/79823d82-b501-47c9-acae-667e192d721b-kube-api-access-gkbv8\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.456816 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456579    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-var-lib-kubelet\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.456816 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456610    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-host\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.456816 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456650    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-var-lib-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.456816 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456667    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysctl-d\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.456816 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456691    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-tmp-dir\") pod \"node-resolver-s2528\" (UID: \"f92ebb16-2530-4f59-98d6-c3c2695ad9ba\") " pod="openshift-dns/node-resolver-s2528"
Oct 30 22:07:41.456816 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456710    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-k8s-cni-cncf-io\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.456816 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456736    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-netns\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.456816 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456768    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-systemd\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.456816 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456792    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsdc4\" (UniqueName: \"kubernetes.io/projected/c0a30486-a1f5-4404-8303-6efed2dad370-kube-api-access-dsdc4\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.457195 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456831    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2188065c-aa72-45ea-a0bb-9665fe8f51a2-agent-certs\") pod \"konnectivity-agent-fgkn6\" (UID: \"2188065c-aa72-45ea-a0bb-9665fe8f51a2\") " pod="kube-system/konnectivity-agent-fgkn6"
Oct 30 22:07:41.457195 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456858    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7048b86e-5ca2-46c8-8b2d-79116829faf2-host-slash\") pod \"iptables-alerter-c5whj\" (UID: \"7048b86e-5ca2-46c8-8b2d-79116829faf2\") " pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:07:41.457195 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456881    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-cnibin\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.457195 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456920    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:41.457195 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456943    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-system-cni-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.457195 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.456984    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgr7f\" (UniqueName: \"kubernetes.io/projected/41dcf151-1d72-4978-b0aa-2046f213cb7e-kube-api-access-rgr7f\") pod \"node-ca-bc87g\" (UID: \"41dcf151-1d72-4978-b0aa-2046f213cb7e\") " pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:07:41.457195 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457011    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.457195 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457035    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-cni-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.457195 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457063    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-cnibin\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.457195 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457080    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-etc-kubernetes\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.457195 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457094    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-run-ovn-kubernetes\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.457195 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457108    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.457195 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457123    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-etc-selinux\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.457195 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457145    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.457195 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457176    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-ovnkube-config\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.457195 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457201    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-run-netns\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.457859 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457225    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-ovnkube-script-lib\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.457859 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457246    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-sys\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.457859 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457268    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.457859 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457282    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgn2d\" (UniqueName: \"kubernetes.io/projected/bec12d9c-76a5-463a-991e-ca82c18fccd2-kube-api-access-jgn2d\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.457859 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457305    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-slash\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.457859 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457358    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-modprobe-d\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.457859 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457399    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysconfig\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.457859 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457422    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-kubernetes\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.457859 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457448    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0a30486-a1f5-4404-8303-6efed2dad370-etc-tuned\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.457859 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457472    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-system-cni-dir\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.457859 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457500    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93f981a6-4af6-4405-aec6-89e539c42c67-dbus\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:41.457859 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457536    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-device-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.457859 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457571    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2188065c-aa72-45ea-a0bb-9665fe8f51a2-konnectivity-ca\") pod \"konnectivity-agent-fgkn6\" (UID: \"2188065c-aa72-45ea-a0bb-9665fe8f51a2\") " pod="kube-system/konnectivity-agent-fgkn6"
Oct 30 22:07:41.457859 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457596    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7048b86e-5ca2-46c8-8b2d-79116829faf2-iptables-alerter-script\") pod \"iptables-alerter-c5whj\" (UID: \"7048b86e-5ca2-46c8-8b2d-79116829faf2\") " pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:07:41.457859 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457611    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-systemd-units\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.457859 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457629    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-etc-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.458468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457650    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-ovn\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.458468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457673    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysctl-conf\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.458468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457689    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c4bd\" (UniqueName: \"kubernetes.io/projected/3788135a-d420-4738-a292-845c50e486a7-kube-api-access-6c4bd\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:41.458468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457723    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-kubelet\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.458468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457737    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41dcf151-1d72-4978-b0aa-2046f213cb7e-host\") pod \"node-ca-bc87g\" (UID: \"41dcf151-1d72-4978-b0aa-2046f213cb7e\") " pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:07:41.458468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457767    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0a30486-a1f5-4404-8303-6efed2dad370-tmp\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.458468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457794    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-registration-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.458468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457833    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93f981a6-4af6-4405-aec6-89e539c42c67-kubelet-config\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:41.458468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457877    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.458468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457906    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-lib-modules\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.458468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457921    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-hostroot\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.458468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457943    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-run\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.458468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457961    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4s4h\" (UniqueName: \"kubernetes.io/projected/7048b86e-5ca2-46c8-8b2d-79116829faf2-kube-api-access-p4s4h\") pod \"iptables-alerter-c5whj\" (UID: \"7048b86e-5ca2-46c8-8b2d-79116829faf2\") " pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:07:41.458468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.457986    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-cni-bin\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.458468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.458020    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-conf-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.458468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.458068    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:41.458468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.458103    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-log-socket\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.458948 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.458141    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-socket-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.458948 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.458169    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-socket-dir-parent\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.458948 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.458194    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4ww6\" (UniqueName: \"kubernetes.io/projected/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-kube-api-access-p4ww6\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.458948 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.458214    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-env-overrides\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.496615 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.496582    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 22:07:41.496615 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.496598    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 22:07:41.496615 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.496584    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 22:07:41.497391 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.497366    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dl5xt\""
Oct 30 22:07:41.497551 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.497366    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tmrfp\""
Oct 30 22:07:41.497551 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.497525    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2djd4\""
Oct 30 22:07:41.497827 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.497807    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2wgjn\""
Oct 30 22:07:41.497956 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.497859    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 22:07:41.500701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.500192    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 22:07:41.500701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.500248    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:07:41.500701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.500352    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 22:07:41.500701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.500382    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-486qv\""
Oct 30 22:07:41.500701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.500408    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 22:07:41.500701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.500422    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-mmvq7\""
Oct 30 22:07:41.500701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.500443    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-78xqf\""
Oct 30 22:07:41.500701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.500455    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 22:07:41.500701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.500464    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 22:07:41.500701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.500463    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 22:07:41.500701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.500488    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 22:07:41.500701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.500500    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 22:07:41.500701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.500528    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 22:07:41.500701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.500530    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 22:07:41.500701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.500560    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 22:07:41.500701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.500603    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 22:07:41.500701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.500656    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pxb8w"
Oct 30 22:07:41.559147 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559118    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-systemd\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.559266 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559153    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-hosts-file\") pod \"node-resolver-s2528\" (UID: \"f92ebb16-2530-4f59-98d6-c3c2695ad9ba\") " pod="openshift-dns/node-resolver-s2528"
Oct 30 22:07:41.559266 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559175    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-kubelet\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.559266 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559198    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-daemon-config\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.559266 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559220    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-multus-certs\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.559266 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559223    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-systemd\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.559266 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559232    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-hosts-file\") pod \"node-resolver-s2528\" (UID: \"f92ebb16-2530-4f59-98d6-c3c2695ad9ba\") " pod="openshift-dns/node-resolver-s2528"
Oct 30 22:07:41.559266 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559241    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-sys-fs\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.559266 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559242    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-kubelet\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.559266 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559262    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nskz5\" (UniqueName: \"kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5\") pod \"network-check-target-7h6kb\" (UID: \"7c1575a8-9288-4779-aab6-e5409b097e9f\") " pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:07:41.559715 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559280    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-multus-certs\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.559715 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559285    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-node-log\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.559715 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559307    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-cni-bin\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.559715 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559348    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-node-log\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.559715 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559349    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/79823d82-b501-47c9-acae-667e192d721b-ovn-node-metrics-cert\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.559715 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559309    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-sys-fs\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.559715 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559374    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbv8\" (UniqueName: \"kubernetes.io/projected/79823d82-b501-47c9-acae-667e192d721b-kube-api-access-gkbv8\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.559715 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559386    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-cni-bin\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.559715 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559496    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-var-lib-kubelet\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.559715 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559613    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-var-lib-kubelet\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.559715 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559613    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-host\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.559715 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559626    2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 22:07:41.559715 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559640    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-var-lib-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.559715 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559656    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysctl-d\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.559715 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559663    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-host\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.559715 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559710    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-var-lib-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559714    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-tmp-dir\") pod \"node-resolver-s2528\" (UID: \"f92ebb16-2530-4f59-98d6-c3c2695ad9ba\") " pod="openshift-dns/node-resolver-s2528"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559745    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-daemon-config\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559786    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-k8s-cni-cncf-io\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559806    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysctl-d\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559816    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-netns\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559855    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-k8s-cni-cncf-io\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559866    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-netns\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559888    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-systemd\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559934    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsdc4\" (UniqueName: \"kubernetes.io/projected/c0a30486-a1f5-4404-8303-6efed2dad370-kube-api-access-dsdc4\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559962    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-tmp-dir\") pod \"node-resolver-s2528\" (UID: \"f92ebb16-2530-4f59-98d6-c3c2695ad9ba\") " pod="openshift-dns/node-resolver-s2528"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559974    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2188065c-aa72-45ea-a0bb-9665fe8f51a2-agent-certs\") pod \"konnectivity-agent-fgkn6\" (UID: \"2188065c-aa72-45ea-a0bb-9665fe8f51a2\") " pod="kube-system/konnectivity-agent-fgkn6"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.559986    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-systemd\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560024    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7048b86e-5ca2-46c8-8b2d-79116829faf2-host-slash\") pod \"iptables-alerter-c5whj\" (UID: \"7048b86e-5ca2-46c8-8b2d-79116829faf2\") " pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560136    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-cnibin\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560156    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7048b86e-5ca2-46c8-8b2d-79116829faf2-host-slash\") pod \"iptables-alerter-c5whj\" (UID: \"7048b86e-5ca2-46c8-8b2d-79116829faf2\") " pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560161    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560186    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-system-cni-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.560460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560197    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-cnibin\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.561309 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560211    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgr7f\" (UniqueName: \"kubernetes.io/projected/41dcf151-1d72-4978-b0aa-2046f213cb7e-kube-api-access-rgr7f\") pod \"node-ca-bc87g\" (UID: \"41dcf151-1d72-4978-b0aa-2046f213cb7e\") " pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:07:41.561309 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560238    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.561309 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560261    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-system-cni-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.561309 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560281    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-cni-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.561309 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:41.560294    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:07:41.561309 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560319    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-cnibin\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.561309 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560391    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-etc-kubernetes\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.561309 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:41.560422    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs podName:3788135a-d420-4738-a292-845c50e486a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:42.060401162 +0000 UTC m=+2.269033844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs") pod "network-metrics-daemon-g5pws" (UID: "3788135a-d420-4738-a292-845c50e486a7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:07:41.561309 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560421    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-cni-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.561309 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560442    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-run-ovn-kubernetes\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.561309 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560447    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-etc-kubernetes\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.561309 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560471    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.561309 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560476    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-cnibin\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.561309 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560508    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.561309 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560508    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-etc-selinux\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.561309 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560530    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.561309 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560550    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-ovnkube-config\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.562375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560569    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-run-netns\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.562375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560569    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-etc-selinux\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.562375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560582    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-ovnkube-script-lib\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.562375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560477    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-run-ovn-kubernetes\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.562375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560597    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-sys\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.562375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560621    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-run-netns\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.562375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560663    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-sys\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.562375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560926    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.562375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560619    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.562375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560960    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.562375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.560965    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgn2d\" (UniqueName: \"kubernetes.io/projected/bec12d9c-76a5-463a-991e-ca82c18fccd2-kube-api-access-jgn2d\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.562375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561007    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-slash\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.562375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561030    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-modprobe-d\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.562375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561055    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysconfig\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.562375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561066    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.562375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561076    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-kubernetes\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.562375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561099    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0a30486-a1f5-4404-8303-6efed2dad370-etc-tuned\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.563213 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561123    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-system-cni-dir\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.563213 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561144    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-slash\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.563213 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561181    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93f981a6-4af6-4405-aec6-89e539c42c67-dbus\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:41.563213 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561183    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-ovnkube-config\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.563213 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561192    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-ovnkube-script-lib\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.563213 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561196    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-kubernetes\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.563213 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561206    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-device-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.563213 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561241    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-system-cni-dir\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.563213 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561247    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysconfig\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.563213 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561258    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-device-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.563213 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561275    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-modprobe-d\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.563213 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561290    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93f981a6-4af6-4405-aec6-89e539c42c67-dbus\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:41.563213 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561307    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2188065c-aa72-45ea-a0bb-9665fe8f51a2-konnectivity-ca\") pod \"konnectivity-agent-fgkn6\" (UID: \"2188065c-aa72-45ea-a0bb-9665fe8f51a2\") " pod="kube-system/konnectivity-agent-fgkn6"
Oct 30 22:07:41.563213 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561355    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7048b86e-5ca2-46c8-8b2d-79116829faf2-iptables-alerter-script\") pod \"iptables-alerter-c5whj\" (UID: \"7048b86e-5ca2-46c8-8b2d-79116829faf2\") " pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:07:41.563213 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561380    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-systemd-units\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.563213 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561403    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-etc-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.563213 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561426    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-ovn\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561449    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysctl-conf\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561484    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-systemd-units\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561477    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c4bd\" (UniqueName: \"kubernetes.io/projected/3788135a-d420-4738-a292-845c50e486a7-kube-api-access-6c4bd\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561541    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-kubelet\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561566    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41dcf151-1d72-4978-b0aa-2046f213cb7e-host\") pod \"node-ca-bc87g\" (UID: \"41dcf151-1d72-4978-b0aa-2046f213cb7e\") " pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561571    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-etc-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561589    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0a30486-a1f5-4404-8303-6efed2dad370-tmp\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561612    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-registration-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561615    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-kubelet\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561633    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93f981a6-4af6-4405-aec6-89e539c42c67-kubelet-config\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561659    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561700    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-lib-modules\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561723    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-hostroot\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561721    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysctl-conf\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561744    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-run\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561766    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4s4h\" (UniqueName: \"kubernetes.io/projected/7048b86e-5ca2-46c8-8b2d-79116829faf2-kube-api-access-p4s4h\") pod \"iptables-alerter-c5whj\" (UID: \"7048b86e-5ca2-46c8-8b2d-79116829faf2\") " pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561795    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-cni-bin\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.563873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561822    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2188065c-aa72-45ea-a0bb-9665fe8f51a2-konnectivity-ca\") pod \"konnectivity-agent-fgkn6\" (UID: \"2188065c-aa72-45ea-a0bb-9665fe8f51a2\") " pod="kube-system/konnectivity-agent-fgkn6"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561822    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-conf-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561865    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561865    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7048b86e-5ca2-46c8-8b2d-79116829faf2-iptables-alerter-script\") pod \"iptables-alerter-c5whj\" (UID: \"7048b86e-5ca2-46c8-8b2d-79116829faf2\") " pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561884    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561913    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-log-socket\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561940    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-socket-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561966    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-socket-dir-parent\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561969    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-lib-modules\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561986    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-registration-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561541    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-ovn\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.561991    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4ww6\" (UniqueName: \"kubernetes.io/projected/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-kube-api-access-p4ww6\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562011    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-hostroot\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562031    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-env-overrides\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562048    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-log-socket\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562056    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-run\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562100    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-socket-dir-parent\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562144    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41dcf151-1d72-4978-b0aa-2046f213cb7e-host\") pod \"node-ca-bc87g\" (UID: \"41dcf151-1d72-4978-b0aa-2046f213cb7e\") " pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:07:41.564396 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562175    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-conf-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.564857 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562177    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93f981a6-4af6-4405-aec6-89e539c42c67-kubelet-config\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:41.564857 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562144    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-cni-bin\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.564857 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562056    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-cni-binary-copy\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.564857 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562224    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-cni-binary-copy\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.564857 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562251    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-cni-multus\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.564857 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562278    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-cni-netd\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.564857 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:41.562298    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:07:41.564857 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562302    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.564857 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562326    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-socket-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.564857 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562353    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2q5s\" (UniqueName: \"kubernetes.io/projected/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-kube-api-access-j2q5s\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.564857 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:41.562374    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret podName:93f981a6-4af6-4405-aec6-89e539c42c67 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:42.062358251 +0000 UTC m=+2.270990918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret") pod "global-pull-secret-syncer-xtgwp" (UID: "93f981a6-4af6-4405-aec6-89e539c42c67") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:07:41.564857 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562416    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xllgn\" (UniqueName: \"kubernetes.io/projected/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-kube-api-access-xllgn\") pod \"node-resolver-s2528\" (UID: \"f92ebb16-2530-4f59-98d6-c3c2695ad9ba\") " pod="openshift-dns/node-resolver-s2528"
Oct 30 22:07:41.564857 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562442    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41dcf151-1d72-4978-b0aa-2046f213cb7e-serviceca\") pod \"node-ca-bc87g\" (UID: \"41dcf151-1d72-4978-b0aa-2046f213cb7e\") " pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:07:41.564857 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562522    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-cni-netd\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.564857 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562530    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-cni-binary-copy\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.564857 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562645    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.564857 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562694    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-cni-multus\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.565356 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562819    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-cni-binary-copy\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.565356 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562864    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-env-overrides\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.565356 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.562905    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41dcf151-1d72-4978-b0aa-2046f213cb7e-serviceca\") pod \"node-ca-bc87g\" (UID: \"41dcf151-1d72-4978-b0aa-2046f213cb7e\") " pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:07:41.565356 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.563537    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/79823d82-b501-47c9-acae-667e192d721b-ovn-node-metrics-cert\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.565356 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.563683    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0a30486-a1f5-4404-8303-6efed2dad370-etc-tuned\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.565356 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.563874    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2188065c-aa72-45ea-a0bb-9665fe8f51a2-agent-certs\") pod \"konnectivity-agent-fgkn6\" (UID: \"2188065c-aa72-45ea-a0bb-9665fe8f51a2\") " pod="kube-system/konnectivity-agent-fgkn6"
Oct 30 22:07:41.565356 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.564379    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0a30486-a1f5-4404-8303-6efed2dad370-tmp\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.583852 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.583832    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:07:41.619394 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:41.619364    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:07:41.619394 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:41.619392    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:07:41.619598 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:41.619406    2572 projected.go:196] Error preparing data for projected volume kube-api-access-nskz5 for pod openshift-network-diagnostics/network-check-target-7h6kb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:07:41.619598 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:41.619483    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5 podName:7c1575a8-9288-4779-aab6-e5409b097e9f nodeName:}" failed. No retries permitted until 2026-10-30 22:07:42.119463649 +0000 UTC m=+2.328096320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nskz5" (UniqueName: "kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5") pod "network-check-target-7h6kb" (UID: "7c1575a8-9288-4779-aab6-e5409b097e9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:07:41.621840 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.621815    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkbv8\" (UniqueName: \"kubernetes.io/projected/79823d82-b501-47c9-acae-667e192d721b-kube-api-access-gkbv8\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.621940 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.621868    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgr7f\" (UniqueName: \"kubernetes.io/projected/41dcf151-1d72-4978-b0aa-2046f213cb7e-kube-api-access-rgr7f\") pod \"node-ca-bc87g\" (UID: \"41dcf151-1d72-4978-b0aa-2046f213cb7e\") " pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:07:41.622252 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.622235    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgn2d\" (UniqueName: \"kubernetes.io/projected/bec12d9c-76a5-463a-991e-ca82c18fccd2-kube-api-access-jgn2d\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.622543 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.622514    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsdc4\" (UniqueName: \"kubernetes.io/projected/c0a30486-a1f5-4404-8303-6efed2dad370-kube-api-access-dsdc4\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.626888 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.626865    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xllgn\" (UniqueName: \"kubernetes.io/projected/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-kube-api-access-xllgn\") pod \"node-resolver-s2528\" (UID: \"f92ebb16-2530-4f59-98d6-c3c2695ad9ba\") " pod="openshift-dns/node-resolver-s2528"
Oct 30 22:07:41.627450 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.627434    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2q5s\" (UniqueName: \"kubernetes.io/projected/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-kube-api-access-j2q5s\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.627575 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.627559    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c4bd\" (UniqueName: \"kubernetes.io/projected/3788135a-d420-4738-a292-845c50e486a7-kube-api-access-6c4bd\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:41.627714 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.627697    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4ww6\" (UniqueName: \"kubernetes.io/projected/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-kube-api-access-p4ww6\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.627784 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.627768    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4s4h\" (UniqueName: \"kubernetes.io/projected/7048b86e-5ca2-46c8-8b2d-79116829faf2-kube-api-access-p4s4h\") pod \"iptables-alerter-c5whj\" (UID: \"7048b86e-5ca2-46c8-8b2d-79116829faf2\") " pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:07:41.748969 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.748899    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:07:41.750969 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.750946    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:07:41.756406 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.756388    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:07:41.785942 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.785914    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:07:41.786210 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:41.786184    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782d33a65ccf759ca199fdbbd0e7fdf9.slice/crio-3b61d4362c209461fe92640752244e15801e90a93e0168e7597b947a56a628c7 WatchSource:0}: Error finding container 3b61d4362c209461fe92640752244e15801e90a93e0168e7597b947a56a628c7: Status 404 returned error can't find the container with id 3b61d4362c209461fe92640752244e15801e90a93e0168e7597b947a56a628c7
Oct 30 22:07:41.786560 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:41.786536    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa975e9f9cee63edf4d69aef6d66c72.slice/crio-7ed222783bdd148dbf9df772ddd6dae9cb919c817293fa2dc74f531b1e2c093a WatchSource:0}: Error finding container 7ed222783bdd148dbf9df772ddd6dae9cb919c817293fa2dc74f531b1e2c093a: Status 404 returned error can't find the container with id 7ed222783bdd148dbf9df772ddd6dae9cb919c817293fa2dc74f531b1e2c093a
Oct 30 22:07:41.787292 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:41.787276    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41dcf151_1d72_4978_b0aa_2046f213cb7e.slice/crio-33a5dce8aae11c140624982e5f02ee86456363d0feae26297084558448dd7501 WatchSource:0}: Error finding container 33a5dce8aae11c140624982e5f02ee86456363d0feae26297084558448dd7501: Status 404 returned error can't find the container with id 33a5dce8aae11c140624982e5f02ee86456363d0feae26297084558448dd7501
Oct 30 22:07:41.787960 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.787933    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s2528"
Oct 30 22:07:41.788079 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:41.788063    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7048b86e_5ca2_46c8_8b2d_79116829faf2.slice/crio-1f1cdb7881fbb080453f7bb6c53fbac80537ed17128b473b464984c2a6e4db5c WatchSource:0}: Error finding container 1f1cdb7881fbb080453f7bb6c53fbac80537ed17128b473b464984c2a6e4db5c: Status 404 returned error can't find the container with id 1f1cdb7881fbb080453f7bb6c53fbac80537ed17128b473b464984c2a6e4db5c
Oct 30 22:07:41.788858 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:41.788843    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79823d82_b501_47c9_acae_667e192d721b.slice/crio-b0fe66d1f732276bd45a6520e5b59a121b279c81db215d74ca0ef78832325c8b WatchSource:0}: Error finding container b0fe66d1f732276bd45a6520e5b59a121b279c81db215d74ca0ef78832325c8b: Status 404 returned error can't find the container with id b0fe66d1f732276bd45a6520e5b59a121b279c81db215d74ca0ef78832325c8b
Oct 30 22:07:41.793040 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.793017    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:07:41.798432 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:41.798413    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec12d9c_76a5_463a_991e_ca82c18fccd2.slice/crio-619bd399d0079020184e9e977bac7ce21130ffd14b03696dd52ce4a212f73aa0 WatchSource:0}: Error finding container 619bd399d0079020184e9e977bac7ce21130ffd14b03696dd52ce4a212f73aa0: Status 404 returned error can't find the container with id 619bd399d0079020184e9e977bac7ce21130ffd14b03696dd52ce4a212f73aa0
Oct 30 22:07:41.799459 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:41.799441    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92ebb16_2530_4f59_98d6_c3c2695ad9ba.slice/crio-7d137caaea3f768787aeb03e28ac7f21294d6e608030e0c363d6939b4c46519f WatchSource:0}: Error finding container 7d137caaea3f768787aeb03e28ac7f21294d6e608030e0c363d6939b4c46519f: Status 404 returned error can't find the container with id 7d137caaea3f768787aeb03e28ac7f21294d6e608030e0c363d6939b4c46519f
Oct 30 22:07:41.805825 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.805805    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2ptq8"
Oct 30 22:07:41.814535 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:41.814510    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d3f83c_7808_4a9a_84d1_d9428d5bd592.slice/crio-93cebfd8e1701b6f66ce8b91ee9f0e4ad453336d4fdee5ac05f97d618e9607f1 WatchSource:0}: Error finding container 93cebfd8e1701b6f66ce8b91ee9f0e4ad453336d4fdee5ac05f97d618e9607f1: Status 404 returned error can't find the container with id 93cebfd8e1701b6f66ce8b91ee9f0e4ad453336d4fdee5ac05f97d618e9607f1
Oct 30 22:07:41.824082 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.824064    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:07:41.832877 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:41.832859    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a30486_a1f5_4404_8303_6efed2dad370.slice/crio-0ea2114c2a1638ba13f2d27656c237e2370c647edb73e1b09fc9f71ad0277c4c WatchSource:0}: Error finding container 0ea2114c2a1638ba13f2d27656c237e2370c647edb73e1b09fc9f71ad0277c4c: Status 404 returned error can't find the container with id 0ea2114c2a1638ba13f2d27656c237e2370c647edb73e1b09fc9f71ad0277c4c
Oct 30 22:07:41.850114 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.850089    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:07:41.851211 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.851182    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fgkn6"
Oct 30 22:07:41.860042 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:41.860021    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac726c56_0d31_4b0d_ae06_e547b7ba1dbd.slice/crio-33577b627b4a1735308d81e8e822ca8e9bbf7a1d1c2f3477dc5c6d6aa34ca360 WatchSource:0}: Error finding container 33577b627b4a1735308d81e8e822ca8e9bbf7a1d1c2f3477dc5c6d6aa34ca360: Status 404 returned error can't find the container with id 33577b627b4a1735308d81e8e822ca8e9bbf7a1d1c2f3477dc5c6d6aa34ca360
Oct 30 22:07:41.862866 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:07:41.862843    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2188065c_aa72_45ea_a0bb_9665fe8f51a2.slice/crio-91f886b0105538a43484d74538321d69e1c80487ecb28ab205584d477e690228 WatchSource:0}: Error finding container 91f886b0105538a43484d74538321d69e1c80487ecb28ab205584d477e690228: Status 404 returned error can't find the container with id 91f886b0105538a43484d74538321d69e1c80487ecb28ab205584d477e690228
Oct 30 22:07:41.864691 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:41.864670    2572 kuberuntime_manager.go:1707] "Unhandled Error" err="container konnectivity-agent start failed in pod konnectivity-agent-fgkn6_kube-system(2188065c-aa72-45ea-a0bb-9665fe8f51a2): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Oct 30 22:07:41.864769 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:41.864712    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ErrImagePull: \"pull QPS exceeded\"" pod="kube-system/konnectivity-agent-fgkn6" podUID="2188065c-aa72-45ea-a0bb-9665fe8f51a2"
Oct 30 22:07:41.961632 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:41.961607    2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:07:42.066424 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:42.066349    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:42.066424 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:42.066405    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:42.066581 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:42.066465    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:07:42.066581 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:42.066490    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:07:42.066581 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:42.066522    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret podName:93f981a6-4af6-4405-aec6-89e539c42c67 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:43.066508528 +0000 UTC m=+3.275141192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret") pod "global-pull-secret-syncer-xtgwp" (UID: "93f981a6-4af6-4405-aec6-89e539c42c67") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:07:42.066581 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:42.066537    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs podName:3788135a-d420-4738-a292-845c50e486a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:43.066529667 +0000 UTC m=+3.275162318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs") pod "network-metrics-daemon-g5pws" (UID: "3788135a-d420-4738-a292-845c50e486a7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:07:42.128873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:42.128842    2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:07:42.166691 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:42.166667    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nskz5\" (UniqueName: \"kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5\") pod \"network-check-target-7h6kb\" (UID: \"7c1575a8-9288-4779-aab6-e5409b097e9f\") " pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:07:42.166794 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:42.166782    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:07:42.166834 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:42.166798    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:07:42.166834 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:42.166806    2572 projected.go:196] Error preparing data for projected volume kube-api-access-nskz5 for pod openshift-network-diagnostics/network-check-target-7h6kb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:07:42.166891 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:42.166845    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5 podName:7c1575a8-9288-4779-aab6-e5409b097e9f nodeName:}" failed. No retries permitted until 2026-10-30 22:07:43.166834393 +0000 UTC m=+3.375467056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nskz5" (UniqueName: "kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5") pod "network-check-target-7h6kb" (UID: "7c1575a8-9288-4779-aab6-e5409b097e9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:07:42.232262 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:42.232214    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp" event={"ID":"79823d82-b501-47c9-acae-667e192d721b","Type":"ContainerStarted","Data":"b0fe66d1f732276bd45a6520e5b59a121b279c81db215d74ca0ef78832325c8b"}
Oct 30 22:07:42.233281 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:42.233264    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fgkn6" event={"ID":"2188065c-aa72-45ea-a0bb-9665fe8f51a2","Type":"ContainerStarted","Data":"91f886b0105538a43484d74538321d69e1c80487ecb28ab205584d477e690228"}
Oct 30 22:07:42.234433 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:42.234410    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl" event={"ID":"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd","Type":"ContainerStarted","Data":"33577b627b4a1735308d81e8e822ca8e9bbf7a1d1c2f3477dc5c6d6aa34ca360"}
Oct 30 22:07:42.234941 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:42.234919    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef3d8a5c5a290fbfb7b3688e9a224e9c47d9f3326f05a0f73577656437e3f406\\\": ErrImagePull: pull QPS exceeded\"" pod="kube-system/konnectivity-agent-fgkn6" podUID="2188065c-aa72-45ea-a0bb-9665fe8f51a2"
Oct 30 22:07:42.235430 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:42.235414    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-np6dm" event={"ID":"c0a30486-a1f5-4404-8303-6efed2dad370","Type":"ContainerStarted","Data":"0ea2114c2a1638ba13f2d27656c237e2370c647edb73e1b09fc9f71ad0277c4c"}
Oct 30 22:07:42.236473 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:42.236455    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2ptq8" event={"ID":"b4d3f83c-7808-4a9a-84d1-d9428d5bd592","Type":"ContainerStarted","Data":"93cebfd8e1701b6f66ce8b91ee9f0e4ad453336d4fdee5ac05f97d618e9607f1"}
Oct 30 22:07:42.237450 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:42.237430    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s2528" event={"ID":"f92ebb16-2530-4f59-98d6-c3c2695ad9ba","Type":"ContainerStarted","Data":"7d137caaea3f768787aeb03e28ac7f21294d6e608030e0c363d6939b4c46519f"}
Oct 30 22:07:42.238411 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:42.238394    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cs9rl" event={"ID":"bec12d9c-76a5-463a-991e-ca82c18fccd2","Type":"ContainerStarted","Data":"619bd399d0079020184e9e977bac7ce21130ffd14b03696dd52ce4a212f73aa0"}
Oct 30 22:07:42.239454 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:42.239437    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c5whj" event={"ID":"7048b86e-5ca2-46c8-8b2d-79116829faf2","Type":"ContainerStarted","Data":"1f1cdb7881fbb080453f7bb6c53fbac80537ed17128b473b464984c2a6e4db5c"}
Oct 30 22:07:42.240475 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:42.240449    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal" event={"ID":"782d33a65ccf759ca199fdbbd0e7fdf9","Type":"ContainerStarted","Data":"3b61d4362c209461fe92640752244e15801e90a93e0168e7597b947a56a628c7"}
Oct 30 22:07:42.241486 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:42.241468    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal" event={"ID":"aaa975e9f9cee63edf4d69aef6d66c72","Type":"ContainerStarted","Data":"7ed222783bdd148dbf9df772ddd6dae9cb919c817293fa2dc74f531b1e2c093a"}
Oct 30 22:07:42.242438 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:42.242413    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bc87g" event={"ID":"41dcf151-1d72-4978-b0aa-2046f213cb7e","Type":"ContainerStarted","Data":"33a5dce8aae11c140624982e5f02ee86456363d0feae26297084558448dd7501"}
Oct 30 22:07:42.504089 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:42.504027    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 22:02:41 +0000 UTC" deadline="2028-05-18 08:43:22.678578123 +0000 UTC"
Oct 30 22:07:42.504089 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:42.504077    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13570h35m40.174503501s"
Oct 30 22:07:43.076672 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:43.076630    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:43.076867 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:43.076707    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:43.076949 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:43.076866    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:07:43.076949 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:43.076931    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs podName:3788135a-d420-4738-a292-845c50e486a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:45.07691244 +0000 UTC m=+5.285545095 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs") pod "network-metrics-daemon-g5pws" (UID: "3788135a-d420-4738-a292-845c50e486a7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:07:43.077064 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:43.077005    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:07:43.077064 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:43.077050    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret podName:93f981a6-4af6-4405-aec6-89e539c42c67 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:45.077028469 +0000 UTC m=+5.285661126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret") pod "global-pull-secret-syncer-xtgwp" (UID: "93f981a6-4af6-4405-aec6-89e539c42c67") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:07:43.180637 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:43.180332    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nskz5\" (UniqueName: \"kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5\") pod \"network-check-target-7h6kb\" (UID: \"7c1575a8-9288-4779-aab6-e5409b097e9f\") " pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:07:43.180637 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:43.180508    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:07:43.180637 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:43.180529    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:07:43.180637 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:43.180542    2572 projected.go:196] Error preparing data for projected volume kube-api-access-nskz5 for pod openshift-network-diagnostics/network-check-target-7h6kb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:07:43.180637 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:43.180598    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5 podName:7c1575a8-9288-4779-aab6-e5409b097e9f nodeName:}" failed. No retries permitted until 2026-10-30 22:07:45.180579311 +0000 UTC m=+5.389211969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nskz5" (UniqueName: "kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5") pod "network-check-target-7h6kb" (UID: "7c1575a8-9288-4779-aab6-e5409b097e9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:07:43.229472 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:43.228930    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:43.229472 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:43.229057    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtgwp" podUID="93f981a6-4af6-4405-aec6-89e539c42c67"
Oct 30 22:07:43.229472 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:43.229136    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:07:43.229472 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:43.229212    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h6kb" podUID="7c1575a8-9288-4779-aab6-e5409b097e9f"
Oct 30 22:07:43.229472 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:43.229274    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:43.229472 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:43.229355    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g5pws" podUID="3788135a-d420-4738-a292-845c50e486a7"
Oct 30 22:07:43.504398 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:43.504281    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 22:02:41 +0000 UTC" deadline="2028-07-02 03:01:05.063232361 +0000 UTC"
Oct 30 22:07:43.504398 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:43.504360    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14644h53m21.558876717s"
Oct 30 22:07:45.102398 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:45.101912    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:45.102398 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:45.102008    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:45.102398 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:45.102160    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:07:45.102398 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:45.102227    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs podName:3788135a-d420-4738-a292-845c50e486a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:49.1022042 +0000 UTC m=+9.310836870 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs") pod "network-metrics-daemon-g5pws" (UID: "3788135a-d420-4738-a292-845c50e486a7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:07:45.102398 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:45.102295    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:07:45.102398 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:45.102331    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret podName:93f981a6-4af6-4405-aec6-89e539c42c67 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:49.10231884 +0000 UTC m=+9.310951508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret") pod "global-pull-secret-syncer-xtgwp" (UID: "93f981a6-4af6-4405-aec6-89e539c42c67") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:07:45.203353 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:45.203261    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nskz5\" (UniqueName: \"kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5\") pod \"network-check-target-7h6kb\" (UID: \"7c1575a8-9288-4779-aab6-e5409b097e9f\") " pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:07:45.203515 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:45.203499    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:07:45.203566 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:45.203525    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:07:45.203566 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:45.203538    2572 projected.go:196] Error preparing data for projected volume kube-api-access-nskz5 for pod openshift-network-diagnostics/network-check-target-7h6kb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:07:45.203630 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:45.203598    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5 podName:7c1575a8-9288-4779-aab6-e5409b097e9f nodeName:}" failed. No retries permitted until 2026-10-30 22:07:49.203577545 +0000 UTC m=+9.412210220 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nskz5" (UniqueName: "kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5") pod "network-check-target-7h6kb" (UID: "7c1575a8-9288-4779-aab6-e5409b097e9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:07:45.229646 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:45.229616    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:45.229792 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:45.229729    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtgwp" podUID="93f981a6-4af6-4405-aec6-89e539c42c67"
Oct 30 22:07:45.229792 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:45.229776    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:07:45.229863 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:45.229822    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h6kb" podUID="7c1575a8-9288-4779-aab6-e5409b097e9f"
Oct 30 22:07:45.229863 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:45.229856    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:45.229928 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:45.229904    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g5pws" podUID="3788135a-d420-4738-a292-845c50e486a7"
Oct 30 22:07:46.280479 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:46.279143    2572 generic.go:356] "Generic (PLEG): container finished" podID="bec12d9c-76a5-463a-991e-ca82c18fccd2" containerID="916d1fb5f87795da6823e1ad7108e2933046929528d2f0d9a3d5ae04ef21c116" exitCode=0
Oct 30 22:07:46.280479 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:46.279399    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cs9rl" event={"ID":"bec12d9c-76a5-463a-991e-ca82c18fccd2","Type":"ContainerDied","Data":"916d1fb5f87795da6823e1ad7108e2933046929528d2f0d9a3d5ae04ef21c116"}
Oct 30 22:07:47.228611 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:47.228525    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:47.228779 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:47.228682    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtgwp" podUID="93f981a6-4af6-4405-aec6-89e539c42c67"
Oct 30 22:07:47.228840 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:47.228817    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:07:47.228938 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:47.228910    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h6kb" podUID="7c1575a8-9288-4779-aab6-e5409b097e9f"
Oct 30 22:07:47.229016 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:47.229002    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:47.229117 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:47.229085    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g5pws" podUID="3788135a-d420-4738-a292-845c50e486a7"
Oct 30 22:07:47.286392 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:47.286336    2572 generic.go:356] "Generic (PLEG): container finished" podID="782d33a65ccf759ca199fdbbd0e7fdf9" containerID="b975cf9bfbace210008b5891dbd017aecf592b9d5e73cce7371970976209cbb0" exitCode=0
Oct 30 22:07:47.286882 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:47.286444    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal" event={"ID":"782d33a65ccf759ca199fdbbd0e7fdf9","Type":"ContainerDied","Data":"b975cf9bfbace210008b5891dbd017aecf592b9d5e73cce7371970976209cbb0"}
Oct 30 22:07:47.294483 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:47.294437    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal" event={"ID":"aaa975e9f9cee63edf4d69aef6d66c72","Type":"ContainerStarted","Data":"241ce13091b06c2bc61beaa452ef0892ce36b94f7d72e42b69fcc30c6f650d31"}
Oct 30 22:07:47.524228 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:47.524145    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal" podStartSLOduration=6.524125927 podStartE2EDuration="6.524125927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:47.523593635 +0000 UTC m=+7.732226311" watchObservedRunningTime="2026-10-30 22:07:47.524125927 +0000 UTC m=+7.732758610"
Oct 30 22:07:48.309606 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:48.309447    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal" event={"ID":"782d33a65ccf759ca199fdbbd0e7fdf9","Type":"ContainerStarted","Data":"de310eb449e62d2ff9b2677c39ae898ce7c7f25bce046f905921cf0928379996"}
Oct 30 22:07:48.319057 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:48.319031    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bc87g" event={"ID":"41dcf151-1d72-4978-b0aa-2046f213cb7e","Type":"ContainerStarted","Data":"0a8eedae20a94a0778b67d04f51b33987529b4e08ad159d5da7910fde03e2ec1"}
Oct 30 22:07:48.541280 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:48.541225    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal" podStartSLOduration=7.541204981 podStartE2EDuration="7.541204981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:48.426121127 +0000 UTC m=+8.634753801" watchObservedRunningTime="2026-10-30 22:07:48.541204981 +0000 UTC m=+8.749837656"
Oct 30 22:07:49.136008 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:49.135971    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:49.136149 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:49.136037    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:49.136188 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:49.136146    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:07:49.136220 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:49.136193    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs podName:3788135a-d420-4738-a292-845c50e486a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:57.136181084 +0000 UTC m=+17.344813748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs") pod "network-metrics-daemon-g5pws" (UID: "3788135a-d420-4738-a292-845c50e486a7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:07:49.136532 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:49.136513    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:07:49.136607 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:49.136557    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret podName:93f981a6-4af6-4405-aec6-89e539c42c67 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:57.136547449 +0000 UTC m=+17.345180113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret") pod "global-pull-secret-syncer-xtgwp" (UID: "93f981a6-4af6-4405-aec6-89e539c42c67") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:07:49.229072 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:49.229003    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:49.229214 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:49.229140    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtgwp" podUID="93f981a6-4af6-4405-aec6-89e539c42c67"
Oct 30 22:07:49.229214 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:49.229199    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:07:49.229279 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:49.229266    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h6kb" podUID="7c1575a8-9288-4779-aab6-e5409b097e9f"
Oct 30 22:07:49.229320 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:49.229310    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:49.229410 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:49.229396    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g5pws" podUID="3788135a-d420-4738-a292-845c50e486a7"
Oct 30 22:07:49.236811 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:49.236781    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nskz5\" (UniqueName: \"kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5\") pod \"network-check-target-7h6kb\" (UID: \"7c1575a8-9288-4779-aab6-e5409b097e9f\") " pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:07:49.237011 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:49.236995    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:07:49.237101 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:49.237020    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:07:49.237101 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:49.237034    2572 projected.go:196] Error preparing data for projected volume kube-api-access-nskz5 for pod openshift-network-diagnostics/network-check-target-7h6kb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:07:49.237101 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:49.237085    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5 podName:7c1575a8-9288-4779-aab6-e5409b097e9f nodeName:}" failed. No retries permitted until 2026-10-30 22:07:57.237068659 +0000 UTC m=+17.445701324 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nskz5" (UniqueName: "kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5") pod "network-check-target-7h6kb" (UID: "7c1575a8-9288-4779-aab6-e5409b097e9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:07:49.327718 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:49.327688    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl" event={"ID":"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd","Type":"ContainerStarted","Data":"98f9738e1114b5461b953c785ddf46999d63ff10133f3d253935de4ec9c9668d"}
Oct 30 22:07:50.695293 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:50.695249    2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 22:07:51.228484 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:51.228453    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:07:51.228643 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:51.228484    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:51.228643 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:51.228484    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:51.228643 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:51.228574    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h6kb" podUID="7c1575a8-9288-4779-aab6-e5409b097e9f"
Oct 30 22:07:51.228740 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:51.228710    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtgwp" podUID="93f981a6-4af6-4405-aec6-89e539c42c67"
Oct 30 22:07:51.228844 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:51.228818    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g5pws" podUID="3788135a-d420-4738-a292-845c50e486a7"
Oct 30 22:07:51.237493 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:51.237404    2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T22:07:50.695272897Z","UUID":"e27f4314-b3ba-48e4-8ac7-c54215f3b905","Handler":null,"Name":"","Endpoint":""}
Oct 30 22:07:51.239280 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:51.239262    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 22:07:51.239420 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:51.239290    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 22:07:51.335315 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:51.335268    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl" event={"ID":"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd","Type":"ContainerStarted","Data":"90f568f36d97a9f30b97f58e4d79c6a590bce71c87f9b5cec66a7fb9991ce188"}
Oct 30 22:07:51.337084 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:51.337058    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2ptq8" event={"ID":"b4d3f83c-7808-4a9a-84d1-d9428d5bd592","Type":"ContainerStarted","Data":"7d648d48e8b6f71cc43a94d9d53baa84dd9430e1aa83d69f1695e86b8115a563"}
Oct 30 22:07:51.338734 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:51.338712    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s2528" event={"ID":"f92ebb16-2530-4f59-98d6-c3c2695ad9ba","Type":"ContainerStarted","Data":"234bef5d230d37ab8be74d2b7906ba5c432772f82ffbcd0748a081cb804d7586"}
Oct 30 22:07:51.340719 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:51.340694    2572 generic.go:356] "Generic (PLEG): container finished" podID="bec12d9c-76a5-463a-991e-ca82c18fccd2" containerID="0977680a73d92dfcf9f648d256685332f7a90f6aa64e89c9a345a22288025b8f" exitCode=0
Oct 30 22:07:51.340829 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:51.340737    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cs9rl" event={"ID":"bec12d9c-76a5-463a-991e-ca82c18fccd2","Type":"ContainerDied","Data":"0977680a73d92dfcf9f648d256685332f7a90f6aa64e89c9a345a22288025b8f"}
Oct 30 22:07:51.456524 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:51.456467    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2ptq8" podStartSLOduration=1.870934525 podStartE2EDuration="10.456449511s" totalImagesPullingTime="8.585514986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:51.456174866 +0000 UTC m=+11.664807540" watchObservedRunningTime="2026-10-30 22:07:51.456449511 +0000 UTC m=+11.665082183"
Oct 30 22:07:51.457388 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:51.457351    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bc87g" podStartSLOduration=5.439287012 podStartE2EDuration="11.457319769s" totalImagesPullingTime="6.018032757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:48.542163485 +0000 UTC m=+8.750796156" watchObservedRunningTime="2026-10-30 22:07:51.457319769 +0000 UTC m=+11.665952444"
Oct 30 22:07:51.571701 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:51.571644    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-s2528" podStartSLOduration=1.582076096 podStartE2EDuration="10.571625052s" totalImagesPullingTime="8.989548956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:51.571222563 +0000 UTC m=+11.779855255" watchObservedRunningTime="2026-10-30 22:07:51.571625052 +0000 UTC m=+11.780257726"
Oct 30 22:07:52.345168 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:52.345130    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl" event={"ID":"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd","Type":"ContainerStarted","Data":"3225b9bef56fd03ce3ad45db0763ef75ef7deb08b94be147a24ca3ab3781b9e5"}
Oct 30 22:07:52.464362 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:52.464300    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl" podStartSLOduration=1.532165591 podStartE2EDuration="11.464280122s" totalImagesPullingTime="9.932114531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:07:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:52.464149136 +0000 UTC m=+12.672781809" watchObservedRunningTime="2026-10-30 22:07:52.464280122 +0000 UTC m=+12.672912817"
Oct 30 22:07:53.229104 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:53.228930    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:53.229242 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:53.228951    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:53.229242 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:53.229173    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtgwp" podUID="93f981a6-4af6-4405-aec6-89e539c42c67"
Oct 30 22:07:53.229318 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:53.228977    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:07:53.229318 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:53.229259    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g5pws" podUID="3788135a-d420-4738-a292-845c50e486a7"
Oct 30 22:07:53.229424 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:53.229317    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h6kb" podUID="7c1575a8-9288-4779-aab6-e5409b097e9f"
Oct 30 22:07:53.349390 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:53.349351    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-np6dm" event={"ID":"c0a30486-a1f5-4404-8303-6efed2dad370","Type":"ContainerStarted","Data":"a00d0a5621e4102b663e5a26a6b3ebe6aa91f159d956b6ac601ba481a90c7356"}
Oct 30 22:07:53.351716 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:53.351686    2572 generic.go:356] "Generic (PLEG): container finished" podID="bec12d9c-76a5-463a-991e-ca82c18fccd2" containerID="fc6548419a124109cb9db87dbda7785c87080549d74af09881993c2f2138acbf" exitCode=0
Oct 30 22:07:53.351933 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:53.351772    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cs9rl" event={"ID":"bec12d9c-76a5-463a-991e-ca82c18fccd2","Type":"ContainerDied","Data":"fc6548419a124109cb9db87dbda7785c87080549d74af09881993c2f2138acbf"}
Oct 30 22:07:53.353871 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:53.353354    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c5whj" event={"ID":"7048b86e-5ca2-46c8-8b2d-79116829faf2","Type":"ContainerStarted","Data":"705493d821f652df59b59a8a3ee4424d46daf27d8ec1b5475b2d5248a84c050c"}
Oct 30 22:07:53.466731 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:53.466676    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-np6dm" podStartSLOduration=1.505250226 podStartE2EDuration="12.466656021s" totalImagesPullingTime="10.961405795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:53.466343357 +0000 UTC m=+13.674976023" watchObservedRunningTime="2026-10-30 22:07:53.466656021 +0000 UTC m=+13.675288697"
Oct 30 22:07:53.581968 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:53.581876    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c5whj" podStartSLOduration=4.497460246 podStartE2EDuration="13.581864725s" totalImagesPullingTime="9.084404479s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:53.58128648 +0000 UTC m=+13.789919154" watchObservedRunningTime="2026-10-30 22:07:53.581864725 +0000 UTC m=+13.790497398"
Oct 30 22:07:55.229473 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:55.229382    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:55.229473 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:55.229427    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:55.230122 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:55.229524    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g5pws" podUID="3788135a-d420-4738-a292-845c50e486a7"
Oct 30 22:07:55.230122 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:55.229607    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtgwp" podUID="93f981a6-4af6-4405-aec6-89e539c42c67"
Oct 30 22:07:55.230122 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:55.229674    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:07:55.230122 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:55.229738    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h6kb" podUID="7c1575a8-9288-4779-aab6-e5409b097e9f"
Oct 30 22:07:55.360934 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:55.360902    2572 generic.go:356] "Generic (PLEG): container finished" podID="bec12d9c-76a5-463a-991e-ca82c18fccd2" containerID="cea493bd953e4555077717d832dbb2d551f2e65bedd9128d0e435d3a57bf7a96" exitCode=0
Oct 30 22:07:55.360934 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:55.360942    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cs9rl" event={"ID":"bec12d9c-76a5-463a-991e-ca82c18fccd2","Type":"ContainerDied","Data":"cea493bd953e4555077717d832dbb2d551f2e65bedd9128d0e435d3a57bf7a96"}
Oct 30 22:07:57.211712 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:57.211677    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:57.212316 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:57.211741    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:57.212316 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:57.211865    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:07:57.212316 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:57.211870    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:07:57.212316 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:57.212262    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs podName:3788135a-d420-4738-a292-845c50e486a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:13.212239955 +0000 UTC m=+33.420872621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs") pod "network-metrics-daemon-g5pws" (UID: "3788135a-d420-4738-a292-845c50e486a7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:07:57.212316 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:57.212288    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret podName:93f981a6-4af6-4405-aec6-89e539c42c67 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:13.212276915 +0000 UTC m=+33.420909572 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret") pod "global-pull-secret-syncer-xtgwp" (UID: "93f981a6-4af6-4405-aec6-89e539c42c67") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:07:57.228432 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:57.228403    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:57.228432 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:57.228428    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:57.228624 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:57.228413    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:07:57.228624 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:57.228499    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g5pws" podUID="3788135a-d420-4738-a292-845c50e486a7"
Oct 30 22:07:57.228624 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:57.228613    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtgwp" podUID="93f981a6-4af6-4405-aec6-89e539c42c67"
Oct 30 22:07:57.228724 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:57.228688    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h6kb" podUID="7c1575a8-9288-4779-aab6-e5409b097e9f"
Oct 30 22:07:57.312996 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:57.312966    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nskz5\" (UniqueName: \"kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5\") pod \"network-check-target-7h6kb\" (UID: \"7c1575a8-9288-4779-aab6-e5409b097e9f\") " pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:07:57.313157 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:57.313130    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:07:57.313157 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:57.313149    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:07:57.313255 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:57.313161    2572 projected.go:196] Error preparing data for projected volume kube-api-access-nskz5 for pod openshift-network-diagnostics/network-check-target-7h6kb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:07:57.313255 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:57.313216    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5 podName:7c1575a8-9288-4779-aab6-e5409b097e9f nodeName:}" failed. No retries permitted until 2026-10-30 22:08:13.313197881 +0000 UTC m=+33.521830554 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nskz5" (UniqueName: "kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5") pod "network-check-target-7h6kb" (UID: "7c1575a8-9288-4779-aab6-e5409b097e9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:07:58.369207 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:58.368891    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fgkn6" event={"ID":"2188065c-aa72-45ea-a0bb-9665fe8f51a2","Type":"ContainerStarted","Data":"28e93cd9610f823829625871f425297dad9891b5807dd1de863ac19424035672"}
Oct 30 22:07:58.371574 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:58.371549    2572 generic.go:356] "Generic (PLEG): container finished" podID="bec12d9c-76a5-463a-991e-ca82c18fccd2" containerID="5465ab5027d6e81a5c3dc426f89ba00960dd4751659748a75449ab0fad9d13dd" exitCode=0
Oct 30 22:07:58.371660 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:58.371637    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cs9rl" event={"ID":"bec12d9c-76a5-463a-991e-ca82c18fccd2","Type":"ContainerDied","Data":"5465ab5027d6e81a5c3dc426f89ba00960dd4751659748a75449ab0fad9d13dd"}
Oct 30 22:07:58.374785 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:58.374766    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp" event={"ID":"79823d82-b501-47c9-acae-667e192d721b","Type":"ContainerStarted","Data":"84e47ad5edfcded23cfeb4245ed36f687d979833aea4cc588d5c6235532be35f"}
Oct 30 22:07:58.374896 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:58.374821    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp" event={"ID":"79823d82-b501-47c9-acae-667e192d721b","Type":"ContainerStarted","Data":"47c1d28e0f0d4c18bdbeec12fe043db3f0da98af4ad1a94e0a9f167f9f55120d"}
Oct 30 22:07:58.374896 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:58.374853    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp" event={"ID":"79823d82-b501-47c9-acae-667e192d721b","Type":"ContainerStarted","Data":"484e83f9bc1661e8ddfe48c18bce7832cbeb3d8af30ce2624b52c30c06a3a306"}
Oct 30 22:07:58.374896 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:58.374882    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp" event={"ID":"79823d82-b501-47c9-acae-667e192d721b","Type":"ContainerStarted","Data":"ba236e53593750dd47f65a0f9b3a1d655161290d6863dc09fafff0e929d4ddfb"}
Oct 30 22:07:58.375003 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:58.374908    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp" event={"ID":"79823d82-b501-47c9-acae-667e192d721b","Type":"ContainerStarted","Data":"71873a7074918947a645f5c3d507cc1f47314b7881f0aed371091718fe8319ea"}
Oct 30 22:07:58.375003 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:58.374933    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp" event={"ID":"79823d82-b501-47c9-acae-667e192d721b","Type":"ContainerStarted","Data":"b325181fd8ba23697f8f3a8908b73ca576cbd2cafdd2a9f4cbf8fc86cf9e0dd9"}
Oct 30 22:07:58.486481 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:58.486441    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fgkn6" podStartSLOduration=1.108613083 podStartE2EDuration="17.486427065s" totalImagesPullingTime="16.377813982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-10-30 22:07:58.485869458 +0000 UTC m=+18.694502131" watchObservedRunningTime="2026-10-30 22:07:58.486427065 +0000 UTC m=+18.695059738"
Oct 30 22:07:59.229393 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:59.229355    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:07:59.229569 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:59.229361    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:07:59.229569 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:59.229462    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtgwp" podUID="93f981a6-4af6-4405-aec6-89e539c42c67"
Oct 30 22:07:59.229569 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:59.229362    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:07:59.229569 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:59.229531    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h6kb" podUID="7c1575a8-9288-4779-aab6-e5409b097e9f"
Oct 30 22:07:59.229722 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:07:59.229601    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g5pws" podUID="3788135a-d420-4738-a292-845c50e486a7"
Oct 30 22:07:59.379463 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:59.379429    2572 generic.go:356] "Generic (PLEG): container finished" podID="bec12d9c-76a5-463a-991e-ca82c18fccd2" containerID="1053e01dd359bb54f56c731349b7ad530e787e8a01f7804f6ab14b2033184fe4" exitCode=0
Oct 30 22:07:59.379922 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:07:59.379476    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cs9rl" event={"ID":"bec12d9c-76a5-463a-991e-ca82c18fccd2","Type":"ContainerDied","Data":"1053e01dd359bb54f56c731349b7ad530e787e8a01f7804f6ab14b2033184fe4"}
Oct 30 22:08:00.384418 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:00.384386    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp" event={"ID":"79823d82-b501-47c9-acae-667e192d721b","Type":"ContainerStarted","Data":"28983e31c780a0c60f767e94603cbac1d923f3097971bcf1fe587373d0fa2603"}
Oct 30 22:08:00.387275 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:00.387256    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cs9rl" event={"ID":"bec12d9c-76a5-463a-991e-ca82c18fccd2","Type":"ContainerStarted","Data":"72cead1b7dea62e9765bc9f634c614af91bd29a2cfbf0975203469888cda36d2"}
Oct 30 22:08:00.508689 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:00.508643    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cs9rl" podStartSLOduration=7.863471455 podStartE2EDuration="20.508627219s" totalImagesPullingTime="12.645155764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:07:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:00.508177425 +0000 UTC m=+20.716810099" watchObservedRunningTime="2026-10-30 22:08:00.508627219 +0000 UTC m=+20.717260016"
Oct 30 22:08:01.229044 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:01.229005    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:01.229044 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:01.229044    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:08:01.229286 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:01.229101    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:08:01.229286 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:01.229115    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g5pws" podUID="3788135a-d420-4738-a292-845c50e486a7"
Oct 30 22:08:01.229286 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:01.229167    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h6kb" podUID="7c1575a8-9288-4779-aab6-e5409b097e9f"
Oct 30 22:08:01.229286 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:01.229244    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtgwp" podUID="93f981a6-4af6-4405-aec6-89e539c42c67"
Oct 30 22:08:02.393368 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:02.393069    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fgkn6"
Oct 30 22:08:02.393850 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:02.393745    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fgkn6"
Oct 30 22:08:02.395112 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:02.395080    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp" event={"ID":"79823d82-b501-47c9-acae-667e192d721b","Type":"ContainerStarted","Data":"fb839dc938eb8dfe439b19ad56586a80ee569f2843dd68a131e114b0f7564d0d"}
Oct 30 22:08:02.395357 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:02.395346    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:02.411399 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:02.411377    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:02.686986 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:02.686895    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp" podStartSLOduration=7.082038563 podStartE2EDuration="22.686882785s" totalImagesPullingTime="15.604844222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:02.686478132 +0000 UTC m=+22.895110806" watchObservedRunningTime="2026-10-30 22:08:02.686882785 +0000 UTC m=+22.895515457"
Oct 30 22:08:03.228732 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:03.228699    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:08:03.228901 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:03.228699    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:08:03.228901 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:03.228821    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtgwp" podUID="93f981a6-4af6-4405-aec6-89e539c42c67"
Oct 30 22:08:03.228901 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:03.228845    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:03.229079 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:03.228946    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h6kb" podUID="7c1575a8-9288-4779-aab6-e5409b097e9f"
Oct 30 22:08:03.229079 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:03.229050    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g5pws" podUID="3788135a-d420-4738-a292-845c50e486a7"
Oct 30 22:08:03.396873 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:03.396844    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:08:03.397391 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:03.397370    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:03.411905 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:03.411884    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:04.399526 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:04.399494    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:08:04.782875 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:04.782792    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7h6kb"]
Oct 30 22:08:04.782875 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:04.782856    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g5pws"]
Oct 30 22:08:04.783110 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:04.782920    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:04.783110 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:04.783014    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g5pws" podUID="3788135a-d420-4738-a292-845c50e486a7"
Oct 30 22:08:04.783110 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:04.783102    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:08:04.783258 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:04.783213    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h6kb" podUID="7c1575a8-9288-4779-aab6-e5409b097e9f"
Oct 30 22:08:04.784526 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:04.784504    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xtgwp"]
Oct 30 22:08:04.784639 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:04.784578    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:08:04.784703 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:04.784642    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtgwp" podUID="93f981a6-4af6-4405-aec6-89e539c42c67"
Oct 30 22:08:05.401512 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:05.401486    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:08:06.228719 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:06.228692    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:06.228880 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:06.228692    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:08:06.228880 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:06.228817    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g5pws" podUID="3788135a-d420-4738-a292-845c50e486a7"
Oct 30 22:08:06.228963 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:06.228876    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h6kb" podUID="7c1575a8-9288-4779-aab6-e5409b097e9f"
Oct 30 22:08:06.672756 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:06.672725    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:06.673130 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:06.672947    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:08:06.682652 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:06.682618    2572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp" podUID="79823d82-b501-47c9-acae-667e192d721b" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 30 22:08:06.691751 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:06.691720    2572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp" podUID="79823d82-b501-47c9-acae-667e192d721b" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 30 22:08:07.228542 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:07.228518    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:08:07.228707 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:07.228614    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtgwp" podUID="93f981a6-4af6-4405-aec6-89e539c42c67"
Oct 30 22:08:08.228591 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:08.228565    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:08:08.229004 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:08.228724    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h6kb" podUID="7c1575a8-9288-4779-aab6-e5409b097e9f"
Oct 30 22:08:08.229004 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:08.228764    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:08.229004 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:08.228850    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g5pws" podUID="3788135a-d420-4738-a292-845c50e486a7"
Oct 30 22:08:08.289320 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:08.289289    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fgkn6"
Oct 30 22:08:08.289475 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:08.289434    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:08:08.289931 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:08.289909    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fgkn6"
Oct 30 22:08:09.229259 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:09.229179    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:08:09.229637 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:09.229281    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtgwp" podUID="93f981a6-4af6-4405-aec6-89e539c42c67"
Oct 30 22:08:10.230496 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.230198    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:10.230864 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.230255    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:08:10.230864 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:10.230632    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g5pws" podUID="3788135a-d420-4738-a292-845c50e486a7"
Oct 30 22:08:10.230864 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:10.230660    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h6kb" podUID="7c1575a8-9288-4779-aab6-e5409b097e9f"
Oct 30 22:08:10.612679 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.612646    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeReady"
Oct 30 22:08:10.612847 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.612762    2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 22:08:10.766845 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.766809    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jktvv"]
Oct 30 22:08:10.825443 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.825403    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6dsmb"]
Oct 30 22:08:10.825586 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.825519    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jktvv"
Oct 30 22:08:10.840194 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.840170    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-79956"]
Oct 30 22:08:10.840297 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.840269    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6dsmb"
Oct 30 22:08:10.853071 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.853051    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"]
Oct 30 22:08:10.853166 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.853152    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-79956"
Oct 30 22:08:10.879075 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.879027    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 22:08:10.879075 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.879071    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 22:08:10.881133 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.881111    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-56d5748b65-v9kvx"]
Oct 30 22:08:10.881343 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.881310    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:10.896965 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.896941    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-652qc"]
Oct 30 22:08:10.897180 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.897033    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:10.912674 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.912652    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtpfd\" (UniqueName: \"kubernetes.io/projected/fcb4abba-e59f-4e64-af78-f1e60697b4b2-kube-api-access-mtpfd\") pod \"kas-connection-checker-685895c7c9-jktvv\" (UID: \"fcb4abba-e59f-4e64-af78-f1e60697b4b2\") " pod="kube-system/kas-connection-checker-685895c7c9-jktvv"
Oct 30 22:08:10.920359 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.920326    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"]
Oct 30 22:08:10.920422 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.920406    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:10.933553 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.933535    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"]
Oct 30 22:08:10.933660 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.933646    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"
Oct 30 22:08:10.934696 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.934676    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 22:08:10.934814 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.934797    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-snjn2\""
Oct 30 22:08:10.946074 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.946056    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"]
Oct 30 22:08:10.946238 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.946172    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"
Oct 30 22:08:10.963895 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.963876    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"]
Oct 30 22:08:10.964015 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.964001    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:10.972919 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.972900    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8k87n"]
Oct 30 22:08:10.973045 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.973016    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:10.979918 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.979895    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 22:08:10.979918 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.979913    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 22:08:10.980192 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.980178    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 22:08:10.980236 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.980178    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 22:08:10.980236 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.980204    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 22:08:10.980236 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.980206    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 22:08:10.980362 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.980245    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 22:08:10.980400 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.980384    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 22:08:10.984414 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.984401    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 22:08:10.987133 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.987118    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"]
Oct 30 22:08:10.987241 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.987226    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8k87n"
Oct 30 22:08:10.995418 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.995402    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bdbcb7997-ggr5g"]
Oct 30 22:08:10.995567 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:10.995522    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:11.009605 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.009590    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5ff86975d4-6xkd8"]
Oct 30 22:08:11.009720 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.009706    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.012969 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.012951    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb5baa5b-c887-42ea-b759-9347ae332051-config\") pod \"service-ca-operator-544d749c8c-js4z7\" (UID: \"bb5baa5b-c887-42ea-b759-9347ae332051\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"
Oct 30 22:08:11.013062 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.012976    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/557fc88c-2102-45ec-adc4-80bae4f140cb-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:11.013062 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.012993    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4254134-79fd-411a-a492-3e620d223ddf-serving-cert\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.013062 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.013027    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-default-certificate\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:11.013182 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.013115    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vss5\" (UniqueName: \"kubernetes.io/projected/bb5baa5b-c887-42ea-b759-9347ae332051-kube-api-access-9vss5\") pod \"service-ca-operator-544d749c8c-js4z7\" (UID: \"bb5baa5b-c887-42ea-b759-9347ae332051\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"
Oct 30 22:08:11.013182 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.013175    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4254134-79fd-411a-a492-3e620d223ddf-tmp\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.013267 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.013253    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-stats-auth\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:11.013301 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.013286    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4254134-79fd-411a-a492-3e620d223ddf-service-ca-bundle\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.013422 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.013408    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb5baa5b-c887-42ea-b759-9347ae332051-serving-cert\") pod \"service-ca-operator-544d749c8c-js4z7\" (UID: \"bb5baa5b-c887-42ea-b759-9347ae332051\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"
Oct 30 22:08:11.013468 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.013437    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:11.013499 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.013487    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:11.013548 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.013511    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:11.013600 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.013587    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtpfd\" (UniqueName: \"kubernetes.io/projected/fcb4abba-e59f-4e64-af78-f1e60697b4b2-kube-api-access-mtpfd\") pod \"kas-connection-checker-685895c7c9-jktvv\" (UID: \"fcb4abba-e59f-4e64-af78-f1e60697b4b2\") " pod="kube-system/kas-connection-checker-685895c7c9-jktvv"
Oct 30 22:08:11.013629 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.013611    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7s7n\" (UniqueName: \"kubernetes.io/projected/b9d3d620-4040-45ca-86da-ce6ba161ba3a-kube-api-access-j7s7n\") pod \"kas-connection-checker-685895c7c9-6dsmb\" (UID: \"b9d3d620-4040-45ca-86da-ce6ba161ba3a\") " pod="kube-system/kas-connection-checker-685895c7c9-6dsmb"
Oct 30 22:08:11.013657 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.013631    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4pmg\" (UniqueName: \"kubernetes.io/projected/557fc88c-2102-45ec-adc4-80bae4f140cb-kube-api-access-h4pmg\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:11.013657 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.013647    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mkwp\" (UniqueName: \"kubernetes.io/projected/ba0fc359-3a19-49ea-b798-2ca0080b30b9-kube-api-access-8mkwp\") pod \"kas-connection-checker-685895c7c9-79956\" (UID: \"ba0fc359-3a19-49ea-b798-2ca0080b30b9\") " pod="kube-system/kas-connection-checker-685895c7c9-79956"
Oct 30 22:08:11.013738 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.013661    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4254134-79fd-411a-a492-3e620d223ddf-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.013738 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.013675    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5cv2\" (UniqueName: \"kubernetes.io/projected/a4254134-79fd-411a-a492-3e620d223ddf-kube-api-access-l5cv2\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.013738 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.013721    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a4254134-79fd-411a-a492-3e620d223ddf-snapshots\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.013854 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.013836    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vppkm\" (UniqueName: \"kubernetes.io/projected/94490976-1f2d-4db5-980a-322cad7c057b-kube-api-access-vppkm\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:11.018573 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.018557    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-f8tgh"]
Oct 30 22:08:11.018653 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.018614    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.026932 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.026915    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jg2dx"]
Oct 30 22:08:11.027024 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.027012    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f8tgh"
Oct 30 22:08:11.029771 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.029754    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 22:08:11.029872 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.029811    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 22:08:11.029872 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.029752    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:08:11.029872 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.029817    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 22:08:11.030018 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.030005    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 22:08:11.030088 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.030070    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:08:11.030142 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.030076    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 22:08:11.030142 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.030129    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:08:11.030142 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.030132    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 22:08:11.030274 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.030129    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 22:08:11.030274 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.030183    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:08:11.030274 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.030204    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 22:08:11.030419 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.030319    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 22:08:11.033284 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.033261    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-svwl7"]
Oct 30 22:08:11.033506 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.033490    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:11.035113 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.035087    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 22:08:11.035206 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.035198    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 22:08:11.035327 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.035311    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-shbx7\""
Oct 30 22:08:11.037157 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.037135    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 22:08:11.037298 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.037144    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 22:08:11.038185 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.038165    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 22:08:11.038275 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.038170    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 22:08:11.038275 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.038205    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 22:08:11.038275 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.038233    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 22:08:11.038275 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.038263    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 22:08:11.038275 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.038265    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-h8kpj\""
Oct 30 22:08:11.038538 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.038284    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wwvsp\""
Oct 30 22:08:11.038538 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.038361    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4zbp8\""
Oct 30 22:08:11.038538 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.038423    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jnhs4\""
Oct 30 22:08:11.038538 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.038450    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-r8xph\""
Oct 30 22:08:11.039501 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039483    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6dsmb"]
Oct 30 22:08:11.039563 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039517    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jktvv"]
Oct 30 22:08:11.039563 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039532    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-svwl7"
Oct 30 22:08:11.039563 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039540    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-79956"]
Oct 30 22:08:11.039563 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039560    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"]
Oct 30 22:08:11.039723 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039484    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-d2r6m\""
Oct 30 22:08:11.039723 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039581    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"]
Oct 30 22:08:11.039723 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039602    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"]
Oct 30 22:08:11.039723 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039629    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56d5748b65-v9kvx"]
Oct 30 22:08:11.039723 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039649    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8k87n"]
Oct 30 22:08:11.039723 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039667    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"]
Oct 30 22:08:11.039723 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039687    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-f8tgh"]
Oct 30 22:08:11.039723 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039706    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jg2dx"]
Oct 30 22:08:11.039994 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039730    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"]
Oct 30 22:08:11.039994 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039749    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"]
Oct 30 22:08:11.039994 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039771    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-svwl7"]
Oct 30 22:08:11.039994 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039790    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-652qc"]
Oct 30 22:08:11.039994 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039807    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bdbcb7997-ggr5g"]
Oct 30 22:08:11.039994 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039705    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:08:11.039994 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039827    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5ff86975d4-6xkd8"]
Oct 30 22:08:11.039994 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.039842    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 22:08:11.047655 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.047639    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-cdzq2\""
Oct 30 22:08:11.047726 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.047715    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 22:08:11.047762 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.047753    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 22:08:11.062426 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.062405    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 22:08:11.062651 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.062634    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 22:08:11.062888 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.062873    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 22:08:11.062956 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.062893    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s8t4z\""
Oct 30 22:08:11.079686 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.079671    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 22:08:11.079760 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.079724    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 22:08:11.079760 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.079733    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 22:08:11.079860 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.079790    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-r9j8h\""
Oct 30 22:08:11.083323 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.083301    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtpfd\" (UniqueName: \"kubernetes.io/projected/fcb4abba-e59f-4e64-af78-f1e60697b4b2-kube-api-access-mtpfd\") pod \"kas-connection-checker-685895c7c9-jktvv\" (UID: \"fcb4abba-e59f-4e64-af78-f1e60697b4b2\") " pod="kube-system/kas-connection-checker-685895c7c9-jktvv"
Oct 30 22:08:11.085919 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.085903    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qzh2l\""
Oct 30 22:08:11.085995 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.085908    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 22:08:11.086364 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.086347    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 22:08:11.092766 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.092750    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 22:08:11.092886 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.092873    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 22:08:11.114771 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.114750    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-bound-sa-token\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.114847 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.114778    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4254134-79fd-411a-a492-3e620d223ddf-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.114847 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.114796    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5cv2\" (UniqueName: \"kubernetes.io/projected/a4254134-79fd-411a-a492-3e620d223ddf-kube-api-access-l5cv2\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.114847 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.114816    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb5baa5b-c887-42ea-b759-9347ae332051-serving-cert\") pod \"service-ca-operator-544d749c8c-js4z7\" (UID: \"bb5baa5b-c887-42ea-b759-9347ae332051\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"
Oct 30 22:08:11.114998 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.114878    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00c23fb6-cbfb-4498-ad17-15998e6d2d66-trusted-ca\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:11.114998 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.114908    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-registry-certificates\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.114998 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.114926    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-trusted-ca\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.114998 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.114973    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4pmg\" (UniqueName: \"kubernetes.io/projected/557fc88c-2102-45ec-adc4-80bae4f140cb-kube-api-access-h4pmg\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:11.115185 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115023    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64d4l\" (UniqueName: \"kubernetes.io/projected/da3555d3-b636-4047-957a-632dd893372e-kube-api-access-64d4l\") pod \"volume-data-source-validator-58c9f5f46d-8k87n\" (UID: \"da3555d3-b636-4047-957a-632dd893372e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8k87n"
Oct 30 22:08:11.115185 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115138    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a4254134-79fd-411a-a492-3e620d223ddf-snapshots\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.115278 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115233    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7n8j\" (UniqueName: \"kubernetes.io/projected/610534ce-c054-41fd-8ab6-b70696b7e360-kube-api-access-v7n8j\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:11.115278 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115268    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.115401 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115289    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trn7v\" (UniqueName: \"kubernetes.io/projected/7d098f8f-7073-4413-9add-d07ef7ce41e1-kube-api-access-trn7v\") pod \"network-check-source-6f78d59f99-f8tgh\" (UID: \"7d098f8f-7073-4413-9add-d07ef7ce41e1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f8tgh"
Oct 30 22:08:11.115401 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115380    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00c23fb6-cbfb-4498-ad17-15998e6d2d66-serving-cert\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:11.115503 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115425    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09b48430-dd49-439d-b317-384408fec509-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp\" (UID: \"09b48430-dd49-439d-b317-384408fec509\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"
Oct 30 22:08:11.115503 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115450    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-bound-sa-token\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.115604 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115545    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-default-certificate\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:11.115662 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115620    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:11.115662 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115639    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a4254134-79fd-411a-a492-3e620d223ddf-snapshots\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.115662 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115653    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-stats-auth\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:11.115800 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115691    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4254134-79fd-411a-a492-3e620d223ddf-service-ca-bundle\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.115800 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115711    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-config-volume\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:11.115800 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115728    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-image-registry-private-configuration\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.115800 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115750    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mkwp\" (UniqueName: \"kubernetes.io/projected/ba0fc359-3a19-49ea-b798-2ca0080b30b9-kube-api-access-8mkwp\") pod \"kas-connection-checker-685895c7c9-79956\" (UID: \"ba0fc359-3a19-49ea-b798-2ca0080b30b9\") " pod="kube-system/kas-connection-checker-685895c7c9-79956"
Oct 30 22:08:11.115974 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115827    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4254134-79fd-411a-a492-3e620d223ddf-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.115974 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115842    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:11.115974 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115940    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8kpl\" (UniqueName: \"kubernetes.io/projected/09b48430-dd49-439d-b317-384408fec509-kube-api-access-f8kpl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp\" (UID: \"09b48430-dd49-439d-b317-384408fec509\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"
Oct 30 22:08:11.116105 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.115989    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwmpt\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-kube-api-access-rwmpt\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.116105 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.116029    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:11.116105 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116027    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-image-registry-private-configuration\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.116105 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116086    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-tmp-dir\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:11.116295 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.116139    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls podName:557fc88c-2102-45ec-adc4-80bae4f140cb nodeName:}" failed. No retries permitted until 2026-10-30 22:08:11.616119363 +0000 UTC m=+31.824752040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-k5qrx" (UID: "557fc88c-2102-45ec-adc4-80bae4f140cb") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:11.116295 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116177    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:11.116295 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116184    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4254134-79fd-411a-a492-3e620d223ddf-service-ca-bundle\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.116295 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116235    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:11.116295 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.116273    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:11.616259813 +0000 UTC m=+31.824892484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : configmap references non-existent config key: service-ca.crt
Oct 30 22:08:11.116555 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116312    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-certificates\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.116555 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.116356    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:08:11.116555 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116358    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:11.116555 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.116421    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:11.616405045 +0000 UTC m=+31.825037708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : secret "router-metrics-certs-default" not found
Oct 30 22:08:11.116555 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116447    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00c23fb6-cbfb-4498-ad17-15998e6d2d66-config\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:11.116555 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116477    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7s7n\" (UniqueName: \"kubernetes.io/projected/b9d3d620-4040-45ca-86da-ce6ba161ba3a-kube-api-access-j7s7n\") pod \"kas-connection-checker-685895c7c9-6dsmb\" (UID: \"b9d3d620-4040-45ca-86da-ce6ba161ba3a\") " pod="kube-system/kas-connection-checker-685895c7c9-6dsmb"
Oct 30 22:08:11.116555 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116531    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:11.116870 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116558    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09b48430-dd49-439d-b317-384408fec509-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp\" (UID: \"09b48430-dd49-439d-b317-384408fec509\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"
Oct 30 22:08:11.116870 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116582    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.116870 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116618    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-installation-pull-secrets\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.116870 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116678    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx7tp\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-kube-api-access-fx7tp\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.116870 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116725    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl8s9\" (UniqueName: \"kubernetes.io/projected/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-kube-api-access-jl8s9\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:11.116870 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116764    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vppkm\" (UniqueName: \"kubernetes.io/projected/94490976-1f2d-4db5-980a-322cad7c057b-kube-api-access-vppkm\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:11.116870 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116791    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-ca-trust-extracted\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.117173 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116870    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb5baa5b-c887-42ea-b759-9347ae332051-config\") pod \"service-ca-operator-544d749c8c-js4z7\" (UID: \"bb5baa5b-c887-42ea-b759-9347ae332051\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"
Oct 30 22:08:11.117173 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116902    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/557fc88c-2102-45ec-adc4-80bae4f140cb-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:11.117173 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116942    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4254134-79fd-411a-a492-3e620d223ddf-serving-cert\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.117173 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116970    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba52011d-6892-4b4d-ac14-07e8be859d9e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:11.117173 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.116997    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8969718f-1dae-4bde-81df-592ee58a1f76-ca-trust-extracted\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.117173 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.117026    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-trusted-ca\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.117173 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.117051    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:11.117173 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.117081    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vss5\" (UniqueName: \"kubernetes.io/projected/bb5baa5b-c887-42ea-b759-9347ae332051-kube-api-access-9vss5\") pod \"service-ca-operator-544d749c8c-js4z7\" (UID: \"bb5baa5b-c887-42ea-b759-9347ae332051\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"
Oct 30 22:08:11.117173 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.117105    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4254134-79fd-411a-a492-3e620d223ddf-tmp\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.117173 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.117136    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8nn2\" (UniqueName: \"kubernetes.io/projected/00c23fb6-cbfb-4498-ad17-15998e6d2d66-kube-api-access-b8nn2\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:11.117173 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.117160    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-installation-pull-secrets\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.117865 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.117413    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb5baa5b-c887-42ea-b759-9347ae332051-config\") pod \"service-ca-operator-544d749c8c-js4z7\" (UID: \"bb5baa5b-c887-42ea-b759-9347ae332051\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"
Oct 30 22:08:11.117865 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.117662    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4254134-79fd-411a-a492-3e620d223ddf-tmp\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.117865 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.117742    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/557fc88c-2102-45ec-adc4-80bae4f140cb-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:11.118176 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.118156    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb5baa5b-c887-42ea-b759-9347ae332051-serving-cert\") pod \"service-ca-operator-544d749c8c-js4z7\" (UID: \"bb5baa5b-c887-42ea-b759-9347ae332051\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"
Oct 30 22:08:11.118525 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.118503    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-stats-auth\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:11.118605 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.118503    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-default-certificate\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:11.119273 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.119255    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4254134-79fd-411a-a492-3e620d223ddf-serving-cert\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.136251 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.136198    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 22:08:11.136251 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.136199    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-kpsh7\""
Oct 30 22:08:11.154178 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.154160    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jktvv"
Oct 30 22:08:11.180357 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.180035    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mkwp\" (UniqueName: \"kubernetes.io/projected/ba0fc359-3a19-49ea-b798-2ca0080b30b9-kube-api-access-8mkwp\") pod \"kas-connection-checker-685895c7c9-79956\" (UID: \"ba0fc359-3a19-49ea-b798-2ca0080b30b9\") " pod="kube-system/kas-connection-checker-685895c7c9-79956"
Oct 30 22:08:11.180357 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.180204    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5cv2\" (UniqueName: \"kubernetes.io/projected/a4254134-79fd-411a-a492-3e620d223ddf-kube-api-access-l5cv2\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.180570 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.180551    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4pmg\" (UniqueName: \"kubernetes.io/projected/557fc88c-2102-45ec-adc4-80bae4f140cb-kube-api-access-h4pmg\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:11.181120 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.181099    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vss5\" (UniqueName: \"kubernetes.io/projected/bb5baa5b-c887-42ea-b759-9347ae332051-kube-api-access-9vss5\") pod \"service-ca-operator-544d749c8c-js4z7\" (UID: \"bb5baa5b-c887-42ea-b759-9347ae332051\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"
Oct 30 22:08:11.181509 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.181492    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7s7n\" (UniqueName: \"kubernetes.io/projected/b9d3d620-4040-45ca-86da-ce6ba161ba3a-kube-api-access-j7s7n\") pod \"kas-connection-checker-685895c7c9-6dsmb\" (UID: \"b9d3d620-4040-45ca-86da-ce6ba161ba3a\") " pod="kube-system/kas-connection-checker-685895c7c9-6dsmb"
Oct 30 22:08:11.184554 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.184536    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vppkm\" (UniqueName: \"kubernetes.io/projected/94490976-1f2d-4db5-980a-322cad7c057b-kube-api-access-vppkm\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:11.190354 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.190323    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-79956"
Oct 30 22:08:11.218437 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218410    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba52011d-6892-4b4d-ac14-07e8be859d9e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:11.218548 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218442    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8969718f-1dae-4bde-81df-592ee58a1f76-ca-trust-extracted\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.218548 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218461    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-trusted-ca\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.218548 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218478    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:11.218548 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218497    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8nn2\" (UniqueName: \"kubernetes.io/projected/00c23fb6-cbfb-4498-ad17-15998e6d2d66-kube-api-access-b8nn2\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:11.218548 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218519    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-installation-pull-secrets\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.218548 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218535    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-bound-sa-token\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.218848 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218564    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00c23fb6-cbfb-4498-ad17-15998e6d2d66-trusted-ca\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:11.218848 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218592    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-registry-certificates\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.218848 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218609    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-trusted-ca\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.218848 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218626    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64d4l\" (UniqueName: \"kubernetes.io/projected/da3555d3-b636-4047-957a-632dd893372e-kube-api-access-64d4l\") pod \"volume-data-source-validator-58c9f5f46d-8k87n\" (UID: \"da3555d3-b636-4047-957a-632dd893372e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8k87n"
Oct 30 22:08:11.218848 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218642    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7n8j\" (UniqueName: \"kubernetes.io/projected/610534ce-c054-41fd-8ab6-b70696b7e360-kube-api-access-v7n8j\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:11.218848 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218664    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.218848 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218680    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trn7v\" (UniqueName: \"kubernetes.io/projected/7d098f8f-7073-4413-9add-d07ef7ce41e1-kube-api-access-trn7v\") pod \"network-check-source-6f78d59f99-f8tgh\" (UID: \"7d098f8f-7073-4413-9add-d07ef7ce41e1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f8tgh"
Oct 30 22:08:11.218848 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218696    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00c23fb6-cbfb-4498-ad17-15998e6d2d66-serving-cert\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:11.218848 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218713    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09b48430-dd49-439d-b317-384408fec509-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp\" (UID: \"09b48430-dd49-439d-b317-384408fec509\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"
Oct 30 22:08:11.218848 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218730    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-bound-sa-token\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.218848 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218752    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkwmz\" (UniqueName: \"kubernetes.io/projected/d81c8448-4580-4174-a7d8-28f5aadf238f-kube-api-access-vkwmz\") pod \"ingress-canary-svwl7\" (UID: \"d81c8448-4580-4174-a7d8-28f5aadf238f\") " pod="openshift-ingress-canary/ingress-canary-svwl7"
Oct 30 22:08:11.218848 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218780    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:11.218848 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218797    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-config-volume\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:11.218848 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218818    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-image-registry-private-configuration\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.218848 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218843    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8kpl\" (UniqueName: \"kubernetes.io/projected/09b48430-dd49-439d-b317-384408fec509-kube-api-access-f8kpl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp\" (UID: \"09b48430-dd49-439d-b317-384408fec509\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"
Oct 30 22:08:11.219527 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218863    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwmpt\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-kube-api-access-rwmpt\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.219527 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218880    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-image-registry-private-configuration\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.219527 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218903    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-tmp-dir\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:11.219527 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218919    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d81c8448-4580-4174-a7d8-28f5aadf238f-cert\") pod \"ingress-canary-svwl7\" (UID: \"d81c8448-4580-4174-a7d8-28f5aadf238f\") " pod="openshift-ingress-canary/ingress-canary-svwl7"
Oct 30 22:08:11.219527 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218957    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-certificates\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.219527 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218980    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:11.219527 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.218997    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00c23fb6-cbfb-4498-ad17-15998e6d2d66-config\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:11.219527 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.219012    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:11.219527 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.219009    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8969718f-1dae-4bde-81df-592ee58a1f76-ca-trust-extracted\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.219527 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.219028    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09b48430-dd49-439d-b317-384408fec509-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp\" (UID: \"09b48430-dd49-439d-b317-384408fec509\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"
Oct 30 22:08:11.219527 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.219044    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.219527 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.219065    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-installation-pull-secrets\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.219527 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.219087    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7tp\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-kube-api-access-fx7tp\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.219527 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.219103    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl8s9\" (UniqueName: \"kubernetes.io/projected/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-kube-api-access-jl8s9\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:11.219527 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.219124    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-ca-trust-extracted\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.219527 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.219191    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba52011d-6892-4b4d-ac14-07e8be859d9e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:11.220240 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.219365    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:08:11.220240 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.219406    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-ca-trust-extracted\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.220240 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.219430    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert podName:ba52011d-6892-4b4d-ac14-07e8be859d9e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:11.719411202 +0000 UTC m=+31.928043872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8sk5t" (UID: "ba52011d-6892-4b4d-ac14-07e8be859d9e") : secret "networking-console-plugin-cert" not found
Oct 30 22:08:11.220240 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.219513    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:08:11.220240 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.219587    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls podName:610534ce-c054-41fd-8ab6-b70696b7e360 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:11.719569493 +0000 UTC m=+31.928202154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-v6j52" (UID: "610534ce-c054-41fd-8ab6-b70696b7e360") : secret "samples-operator-tls" not found
Oct 30 22:08:11.220240 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.219602    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-trusted-ca\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.220240 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.219904    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:11.220240 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.219959    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-config-volume\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:11.220240 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.220021    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-certificates\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.220240 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.220096    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:11.220240 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.220108    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdbcb7997-ggr5g: secret "image-registry-tls" not found
Oct 30 22:08:11.220240 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.220150    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls podName:8969718f-1dae-4bde-81df-592ee58a1f76 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:11.720133523 +0000 UTC m=+31.928766174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls") pod "image-registry-bdbcb7997-ggr5g" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76") : secret "image-registry-tls" not found
Oct 30 22:08:11.220240 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.220205    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:08:11.220240 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.220240    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls podName:a4369922-ceac-4aaf-9a8b-35c5a0f27d91 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:11.720229819 +0000 UTC m=+31.928862470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls") pod "dns-default-jg2dx" (UID: "a4369922-ceac-4aaf-9a8b-35c5a0f27d91") : secret "dns-default-metrics-tls" not found
Oct 30 22:08:11.222843 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.220389    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00c23fb6-cbfb-4498-ad17-15998e6d2d66-config\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:11.222843 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.220819    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09b48430-dd49-439d-b317-384408fec509-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp\" (UID: \"09b48430-dd49-439d-b317-384408fec509\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"
Oct 30 22:08:11.222843 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.220993    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:11.222843 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.221016    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ff86975d4-6xkd8: secret "image-registry-tls" not found
Oct 30 22:08:11.222843 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.221068    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls podName:4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d nodeName:}" failed. No retries permitted until 2026-10-30 22:08:11.721052718 +0000 UTC m=+31.929685388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls") pod "image-registry-5ff86975d4-6xkd8" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d") : secret "image-registry-tls" not found
Oct 30 22:08:11.222843 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.221486    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00c23fb6-cbfb-4498-ad17-15998e6d2d66-trusted-ca\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:11.222843 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.221586    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-registry-certificates\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.222843 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.222116    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-trusted-ca\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.222843 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.222236    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-tmp-dir\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:11.224384 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.224305    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09b48430-dd49-439d-b317-384408fec509-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp\" (UID: \"09b48430-dd49-439d-b317-384408fec509\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"
Oct 30 22:08:11.225508 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.225397    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-image-registry-private-configuration\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.225707 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.225671    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-image-registry-private-configuration\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.229650 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.228794    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:08:11.233043 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.231879    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00c23fb6-cbfb-4498-ad17-15998e6d2d66-serving-cert\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:11.233043 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.232680    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-installation-pull-secrets\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.233896 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.233863    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-installation-pull-secrets\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.240608 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.240583    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:11.250974 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.250955    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"
Oct 30 22:08:11.281856 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.281826    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8nn2\" (UniqueName: \"kubernetes.io/projected/00c23fb6-cbfb-4498-ad17-15998e6d2d66-kube-api-access-b8nn2\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:11.282323 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.282293    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-bound-sa-token\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.283795 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.283767    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx7tp\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-kube-api-access-fx7tp\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.284257 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.284226    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64d4l\" (UniqueName: \"kubernetes.io/projected/da3555d3-b636-4047-957a-632dd893372e-kube-api-access-64d4l\") pod \"volume-data-source-validator-58c9f5f46d-8k87n\" (UID: \"da3555d3-b636-4047-957a-632dd893372e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8k87n"
Oct 30 22:08:11.284770 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.284750    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 22:08:11.284863 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.284796    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trn7v\" (UniqueName: \"kubernetes.io/projected/7d098f8f-7073-4413-9add-d07ef7ce41e1-kube-api-access-trn7v\") pod \"network-check-source-6f78d59f99-f8tgh\" (UID: \"7d098f8f-7073-4413-9add-d07ef7ce41e1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f8tgh"
Oct 30 22:08:11.284924 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.284883    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-bound-sa-token\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.286460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.285745    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7n8j\" (UniqueName: \"kubernetes.io/projected/610534ce-c054-41fd-8ab6-b70696b7e360-kube-api-access-v7n8j\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:11.286460 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.286173    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8kpl\" (UniqueName: \"kubernetes.io/projected/09b48430-dd49-439d-b317-384408fec509-kube-api-access-f8kpl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp\" (UID: \"09b48430-dd49-439d-b317-384408fec509\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"
Oct 30 22:08:11.286663 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.286634    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwmpt\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-kube-api-access-rwmpt\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.287393 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.286798    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl8s9\" (UniqueName: \"kubernetes.io/projected/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-kube-api-access-jl8s9\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:11.296119 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.296059    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-f5pfp\""
Oct 30 22:08:11.304246 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.304215    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:11.315879 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.315764    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8k87n"
Oct 30 22:08:11.320111 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.320090    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkwmz\" (UniqueName: \"kubernetes.io/projected/d81c8448-4580-4174-a7d8-28f5aadf238f-kube-api-access-vkwmz\") pod \"ingress-canary-svwl7\" (UID: \"d81c8448-4580-4174-a7d8-28f5aadf238f\") " pod="openshift-ingress-canary/ingress-canary-svwl7"
Oct 30 22:08:11.320196 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.320155    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d81c8448-4580-4174-a7d8-28f5aadf238f-cert\") pod \"ingress-canary-svwl7\" (UID: \"d81c8448-4580-4174-a7d8-28f5aadf238f\") " pod="openshift-ingress-canary/ingress-canary-svwl7"
Oct 30 22:08:11.322638 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.322619    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d81c8448-4580-4174-a7d8-28f5aadf238f-cert\") pod \"ingress-canary-svwl7\" (UID: \"d81c8448-4580-4174-a7d8-28f5aadf238f\") " pod="openshift-ingress-canary/ingress-canary-svwl7"
Oct 30 22:08:11.344810 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.343627    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f8tgh"
Oct 30 22:08:11.383310 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.383262    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkwmz\" (UniqueName: \"kubernetes.io/projected/d81c8448-4580-4174-a7d8-28f5aadf238f-kube-api-access-vkwmz\") pod \"ingress-canary-svwl7\" (UID: \"d81c8448-4580-4174-a7d8-28f5aadf238f\") " pod="openshift-ingress-canary/ingress-canary-svwl7"
Oct 30 22:08:11.474746 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.474723    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6dsmb"
Oct 30 22:08:11.556011 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.555943    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jktvv"]
Oct 30 22:08:11.569111 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.568883    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"
Oct 30 22:08:11.573228 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.573013    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-79956"]
Oct 30 22:08:11.577203 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:08:11.577173    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb4abba_e59f_4e64_af78_f1e60697b4b2.slice/crio-1e059cdea8af69c2f35c6f2c86433cdfc163872947514209381afd25293ce48d WatchSource:0}: Error finding container 1e059cdea8af69c2f35c6f2c86433cdfc163872947514209381afd25293ce48d: Status 404 returned error can't find the container with id 1e059cdea8af69c2f35c6f2c86433cdfc163872947514209381afd25293ce48d
Oct 30 22:08:11.578303 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:08:11.578175    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0fc359_3a19_49ea_b798_2ca0080b30b9.slice/crio-f41d8d95231e5c3bf2162b438bdae00456b0e9b7d93677891f6adff38147e21c WatchSource:0}: Error finding container f41d8d95231e5c3bf2162b438bdae00456b0e9b7d93677891f6adff38147e21c: Status 404 returned error can't find the container with id f41d8d95231e5c3bf2162b438bdae00456b0e9b7d93677891f6adff38147e21c
Oct 30 22:08:11.622920 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.622323    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:11.622920 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.622408    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:11.622920 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.622459    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:11.622920 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.622666    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:08:11.622920 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.622725    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:12.622708106 +0000 UTC m=+32.831340772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : secret "router-metrics-certs-default" not found
Oct 30 22:08:11.622920 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.622789    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:11.622920 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.622825    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls podName:557fc88c-2102-45ec-adc4-80bae4f140cb nodeName:}" failed. No retries permitted until 2026-10-30 22:08:12.622813815 +0000 UTC m=+32.831446479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-k5qrx" (UID: "557fc88c-2102-45ec-adc4-80bae4f140cb") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:11.622920 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.622889    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:12.622879611 +0000 UTC m=+32.831512279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : configmap references non-existent config key: service-ca.crt
Oct 30 22:08:11.633846 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.633744    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-652qc"]
Oct 30 22:08:11.660212 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.660029    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-svwl7"
Oct 30 22:08:11.675668 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.675597    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"]
Oct 30 22:08:11.684897 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:08:11.684868    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5baa5b_c887_42ea_b759_9347ae332051.slice/crio-955876a99ad96b940d6504588c872a5ee075b62107954327eb2b6b1f5a086035 WatchSource:0}: Error finding container 955876a99ad96b940d6504588c872a5ee075b62107954327eb2b6b1f5a086035: Status 404 returned error can't find the container with id 955876a99ad96b940d6504588c872a5ee075b62107954327eb2b6b1f5a086035
Oct 30 22:08:11.719943 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.719920    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8k87n"]
Oct 30 22:08:11.720365 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.720321    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"]
Oct 30 22:08:11.723001 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.722980    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:11.723073 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:08:11.722999    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3555d3_b636_4047_957a_632dd893372e.slice/crio-6dd646d0a376f2b2fcfcdb457563de9213574005035152738337b593023b105c WatchSource:0}: Error finding container 6dd646d0a376f2b2fcfcdb457563de9213574005035152738337b593023b105c: Status 404 returned error can't find the container with id 6dd646d0a376f2b2fcfcdb457563de9213574005035152738337b593023b105c
Oct 30 22:08:11.723073 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.723011    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:11.723145 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.723085    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:11.723145 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.723081    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:11.723145 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.723095    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdbcb7997-ggr5g: secret "image-registry-tls" not found
Oct 30 22:08:11.723145 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.723130    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:08:11.723248 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.723133    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls podName:8969718f-1dae-4bde-81df-592ee58a1f76 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:12.723120923 +0000 UTC m=+32.931753573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls") pod "image-registry-bdbcb7997-ggr5g" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76") : secret "image-registry-tls" not found
Oct 30 22:08:11.723248 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.723174    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:08:11.723248 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.723217    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls podName:a4369922-ceac-4aaf-9a8b-35c5a0f27d91 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:12.723202885 +0000 UTC m=+32.931835542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls") pod "dns-default-jg2dx" (UID: "a4369922-ceac-4aaf-9a8b-35c5a0f27d91") : secret "dns-default-metrics-tls" not found
Oct 30 22:08:11.723248 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.723231    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls podName:610534ce-c054-41fd-8ab6-b70696b7e360 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:12.723225435 +0000 UTC m=+32.931858085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-v6j52" (UID: "610534ce-c054-41fd-8ab6-b70696b7e360") : secret "samples-operator-tls" not found
Oct 30 22:08:11.723381 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.723256    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:11.723381 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.723291    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:11.723381 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.723297    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:11.723381 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.723304    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ff86975d4-6xkd8: secret "image-registry-tls" not found
Oct 30 22:08:11.723381 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.723325    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls podName:4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d nodeName:}" failed. No retries permitted until 2026-10-30 22:08:12.723317993 +0000 UTC m=+32.931950644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls") pod "image-registry-5ff86975d4-6xkd8" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d") : secret "image-registry-tls" not found
Oct 30 22:08:11.723381 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.723372    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:08:11.723630 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:11.723394    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert podName:ba52011d-6892-4b4d-ac14-07e8be859d9e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:12.723388143 +0000 UTC m=+32.932020833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8sk5t" (UID: "ba52011d-6892-4b4d-ac14-07e8be859d9e") : secret "networking-console-plugin-cert" not found
Oct 30 22:08:11.730396 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:08:11.730374    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c23fb6_cbfb_4498_ad17_15998e6d2d66.slice/crio-1f94e750fc006efb749803797e006255ddf0491f5f89e99505b03a099ef8f8b7 WatchSource:0}: Error finding container 1f94e750fc006efb749803797e006255ddf0491f5f89e99505b03a099ef8f8b7: Status 404 returned error can't find the container with id 1f94e750fc006efb749803797e006255ddf0491f5f89e99505b03a099ef8f8b7
Oct 30 22:08:11.762890 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.762772    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-f8tgh"]
Oct 30 22:08:11.767418 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:08:11.767386    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d098f8f_7073_4413_9add_d07ef7ce41e1.slice/crio-c0eada557aeb34035fd734d4b685481c39bebcff32b33ee74351b9513eb3e22a WatchSource:0}: Error finding container c0eada557aeb34035fd734d4b685481c39bebcff32b33ee74351b9513eb3e22a: Status 404 returned error can't find the container with id c0eada557aeb34035fd734d4b685481c39bebcff32b33ee74351b9513eb3e22a
Oct 30 22:08:11.873634 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.873603    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6dsmb"]
Oct 30 22:08:11.876071 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:08:11.876047    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d3d620_4040_45ca_86da_ce6ba161ba3a.slice/crio-4bf351d84cf28fc3d5c700f4d9c5dbe3275988880a55fa29234c6e3e1b99d073 WatchSource:0}: Error finding container 4bf351d84cf28fc3d5c700f4d9c5dbe3275988880a55fa29234c6e3e1b99d073: Status 404 returned error can't find the container with id 4bf351d84cf28fc3d5c700f4d9c5dbe3275988880a55fa29234c6e3e1b99d073
Oct 30 22:08:11.975484 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:11.975457    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"]
Oct 30 22:08:11.978134 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:08:11.978111    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b48430_dd49_439d_b317_384408fec509.slice/crio-1502caa923a2b2ab695fb6b2b2f7ef5d1b4e3b7dc87fba00ba473f880ead9d4a WatchSource:0}: Error finding container 1502caa923a2b2ab695fb6b2b2f7ef5d1b4e3b7dc87fba00ba473f880ead9d4a: Status 404 returned error can't find the container with id 1502caa923a2b2ab695fb6b2b2f7ef5d1b4e3b7dc87fba00ba473f880ead9d4a
Oct 30 22:08:12.062908 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.062876    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-svwl7"]
Oct 30 22:08:12.065929 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:08:12.065900    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81c8448_4580_4174_a7d8_28f5aadf238f.slice/crio-34feb426af22aebf71fd4d331f0f8b048ce191d95ae6ee3c1ae6cc991c4a706a WatchSource:0}: Error finding container 34feb426af22aebf71fd4d331f0f8b048ce191d95ae6ee3c1ae6cc991c4a706a: Status 404 returned error can't find the container with id 34feb426af22aebf71fd4d331f0f8b048ce191d95ae6ee3c1ae6cc991c4a706a
Oct 30 22:08:12.229029 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.228957    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:08:12.229153 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.229140    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:12.286032 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.285987    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 22:08:12.286032 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.285987    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c8f2h\""
Oct 30 22:08:12.286418 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.286049    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xjxjz\""
Oct 30 22:08:12.416303 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.416263    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8k87n" event={"ID":"da3555d3-b636-4047-957a-632dd893372e","Type":"ContainerStarted","Data":"6dd646d0a376f2b2fcfcdb457563de9213574005035152738337b593023b105c"}
Oct 30 22:08:12.417385 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.417352    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-652qc" event={"ID":"a4254134-79fd-411a-a492-3e620d223ddf","Type":"ContainerStarted","Data":"dc134333d3bbaed5e72f2b58f271684b858ac448fa30ce00164b295bfae297f3"}
Oct 30 22:08:12.418610 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.418591    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-79956" event={"ID":"ba0fc359-3a19-49ea-b798-2ca0080b30b9","Type":"ContainerStarted","Data":"d3b4ffe6262e8b198dfb9efc65c4ee3c3d46003a46a1dbac87f23dd7e99ef3e7"}
Oct 30 22:08:12.418707 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.418622    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-79956" event={"ID":"ba0fc359-3a19-49ea-b798-2ca0080b30b9","Type":"ContainerStarted","Data":"f41d8d95231e5c3bf2162b438bdae00456b0e9b7d93677891f6adff38147e21c"}
Oct 30 22:08:12.422124 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.422089    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jktvv" event={"ID":"fcb4abba-e59f-4e64-af78-f1e60697b4b2","Type":"ContainerStarted","Data":"90984f63cfbc0151a8df5ff6a26024cdd1620019378ace81a37a2ab90e5b7df5"}
Oct 30 22:08:12.422219 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.422126    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jktvv" event={"ID":"fcb4abba-e59f-4e64-af78-f1e60697b4b2","Type":"ContainerStarted","Data":"1e059cdea8af69c2f35c6f2c86433cdfc163872947514209381afd25293ce48d"}
Oct 30 22:08:12.423295 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.423275    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp" event={"ID":"09b48430-dd49-439d-b317-384408fec509","Type":"ContainerStarted","Data":"1502caa923a2b2ab695fb6b2b2f7ef5d1b4e3b7dc87fba00ba473f880ead9d4a"}
Oct 30 22:08:12.424740 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.424719    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7" event={"ID":"bb5baa5b-c887-42ea-b759-9347ae332051","Type":"ContainerStarted","Data":"955876a99ad96b940d6504588c872a5ee075b62107954327eb2b6b1f5a086035"}
Oct 30 22:08:12.425793 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.425772    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-svwl7" event={"ID":"d81c8448-4580-4174-a7d8-28f5aadf238f","Type":"ContainerStarted","Data":"34feb426af22aebf71fd4d331f0f8b048ce191d95ae6ee3c1ae6cc991c4a706a"}
Oct 30 22:08:12.426978 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.426957    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6dsmb" event={"ID":"b9d3d620-4040-45ca-86da-ce6ba161ba3a","Type":"ContainerStarted","Data":"b839db5152af04b497144e4674d32a69a43053b0f2a61cf491c6ba0dcfa2b3f7"}
Oct 30 22:08:12.427053 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.426997    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6dsmb" event={"ID":"b9d3d620-4040-45ca-86da-ce6ba161ba3a","Type":"ContainerStarted","Data":"4bf351d84cf28fc3d5c700f4d9c5dbe3275988880a55fa29234c6e3e1b99d073"}
Oct 30 22:08:12.428209 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.428187    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f8tgh" event={"ID":"7d098f8f-7073-4413-9add-d07ef7ce41e1","Type":"ContainerStarted","Data":"c0eada557aeb34035fd734d4b685481c39bebcff32b33ee74351b9513eb3e22a"}
Oct 30 22:08:12.429268 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.429249    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr" event={"ID":"00c23fb6-cbfb-4498-ad17-15998e6d2d66","Type":"ContainerStarted","Data":"1f94e750fc006efb749803797e006255ddf0491f5f89e99505b03a099ef8f8b7"}
Oct 30 22:08:12.630605 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.630569    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:12.630744 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.630622    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:12.630744 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.630639    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:12.630744 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:12.630672    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:12.630744 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:12.630718    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:08:12.630744 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:12.630745    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls podName:557fc88c-2102-45ec-adc4-80bae4f140cb nodeName:}" failed. No retries permitted until 2026-10-30 22:08:14.630726003 +0000 UTC m=+34.839358653 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-k5qrx" (UID: "557fc88c-2102-45ec-adc4-80bae4f140cb") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:12.630932 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:12.630764    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:14.630752587 +0000 UTC m=+34.839385258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : secret "router-metrics-certs-default" not found
Oct 30 22:08:12.630932 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:12.630777    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:14.630770971 +0000 UTC m=+34.839403621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : configmap references non-existent config key: service-ca.crt
Oct 30 22:08:12.731968 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.731926    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:12.732133 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.732002    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:12.732133 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.732044    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:12.732133 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.732105    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:12.732133 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.732123    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:12.732276 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:12.732242    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:12.732276 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:12.732257    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdbcb7997-ggr5g: secret "image-registry-tls" not found
Oct 30 22:08:12.732399 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:12.732315    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls podName:8969718f-1dae-4bde-81df-592ee58a1f76 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:14.732297974 +0000 UTC m=+34.940930640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls") pod "image-registry-bdbcb7997-ggr5g" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76") : secret "image-registry-tls" not found
Oct 30 22:08:12.732399 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:12.732397    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:08:12.732510 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:12.732421    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls podName:610534ce-c054-41fd-8ab6-b70696b7e360 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:14.732414594 +0000 UTC m=+34.941047258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-v6j52" (UID: "610534ce-c054-41fd-8ab6-b70696b7e360") : secret "samples-operator-tls" not found
Oct 30 22:08:12.732510 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:12.732468    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:12.732510 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:12.732476    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ff86975d4-6xkd8: secret "image-registry-tls" not found
Oct 30 22:08:12.732510 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:12.732494    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls podName:4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d nodeName:}" failed. No retries permitted until 2026-10-30 22:08:14.732488337 +0000 UTC m=+34.941121001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls") pod "image-registry-5ff86975d4-6xkd8" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d") : secret "image-registry-tls" not found
Oct 30 22:08:12.732694 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:12.732535    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:08:12.732694 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:12.732556    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert podName:ba52011d-6892-4b4d-ac14-07e8be859d9e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:14.732549079 +0000 UTC m=+34.941181743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8sk5t" (UID: "ba52011d-6892-4b4d-ac14-07e8be859d9e") : secret "networking-console-plugin-cert" not found
Oct 30 22:08:12.732694 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:12.732590    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:08:12.732694 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:12.732608    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls podName:a4369922-ceac-4aaf-9a8b-35c5a0f27d91 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:14.732602551 +0000 UTC m=+34.941235214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls") pod "dns-default-jg2dx" (UID: "a4369922-ceac-4aaf-9a8b-35c5a0f27d91") : secret "dns-default-metrics-tls" not found
Oct 30 22:08:12.961723 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:12.961585    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6dsmb" podStartSLOduration=393.961565151 podStartE2EDuration="6m33.961565151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:12.960376635 +0000 UTC m=+33.169009345" watchObservedRunningTime="2026-10-30 22:08:12.961565151 +0000 UTC m=+33.170197824"
Oct 30 22:08:13.077517 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:13.077396    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-79956" podStartSLOduration=394.077376298 podStartE2EDuration="6m34.077376298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:13.07542645 +0000 UTC m=+33.284059150" watchObservedRunningTime="2026-10-30 22:08:13.077376298 +0000 UTC m=+33.286008972"
Oct 30 22:08:13.196823 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:13.196764    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jktvv" podStartSLOduration=394.196747309 podStartE2EDuration="6m34.196747309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:13.195503262 +0000 UTC m=+33.404135961" watchObservedRunningTime="2026-10-30 22:08:13.196747309 +0000 UTC m=+33.405379982"
Oct 30 22:08:13.239695 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:13.238877    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:13.239695 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:13.238955    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:08:13.239695 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:13.239553    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:08:13.239695 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:13.239625    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs podName:3788135a-d420-4738-a292-845c50e486a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:45.239603288 +0000 UTC m=+65.448235956 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs") pod "network-metrics-daemon-g5pws" (UID: "3788135a-d420-4738-a292-845c50e486a7") : secret "metrics-daemon-secret" not found
Oct 30 22:08:13.242507 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:13.242478    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:08:13.340375 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:13.340310    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nskz5\" (UniqueName: \"kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5\") pod \"network-check-target-7h6kb\" (UID: \"7c1575a8-9288-4779-aab6-e5409b097e9f\") " pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:08:13.355097 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:13.355067    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nskz5\" (UniqueName: \"kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5\") pod \"network-check-target-7h6kb\" (UID: \"7c1575a8-9288-4779-aab6-e5409b097e9f\") " pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:08:13.365117 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:13.364929    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:08:13.455750 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:13.455135    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:08:13.773686 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:13.773606    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xtgwp"]
Oct 30 22:08:13.781138 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:08:13.781108    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f981a6_4af6_4405_aec6_89e539c42c67.slice/crio-35f92cea187b4b04177b4d09b48957700237301a7d7abafbb402a40cd1305055 WatchSource:0}: Error finding container 35f92cea187b4b04177b4d09b48957700237301a7d7abafbb402a40cd1305055: Status 404 returned error can't find the container with id 35f92cea187b4b04177b4d09b48957700237301a7d7abafbb402a40cd1305055
Oct 30 22:08:13.907941 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:13.907851    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7h6kb"]
Oct 30 22:08:13.912542 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:08:13.912503    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1575a8_9288_4779_aab6_e5409b097e9f.slice/crio-931bbe7eba22c56243937d2b9926956457d6ca1dc097a15c502a24c087717a23 WatchSource:0}: Error finding container 931bbe7eba22c56243937d2b9926956457d6ca1dc097a15c502a24c087717a23: Status 404 returned error can't find the container with id 931bbe7eba22c56243937d2b9926956457d6ca1dc097a15c502a24c087717a23
Oct 30 22:08:14.444288 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:14.444197    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7h6kb" event={"ID":"7c1575a8-9288-4779-aab6-e5409b097e9f","Type":"ContainerStarted","Data":"931bbe7eba22c56243937d2b9926956457d6ca1dc097a15c502a24c087717a23"}
Oct 30 22:08:14.446305 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:14.446278    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xtgwp" event={"ID":"93f981a6-4af6-4405-aec6-89e539c42c67","Type":"ContainerStarted","Data":"35f92cea187b4b04177b4d09b48957700237301a7d7abafbb402a40cd1305055"}
Oct 30 22:08:14.449875 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:14.448921    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8k87n" event={"ID":"da3555d3-b636-4047-957a-632dd893372e","Type":"ContainerStarted","Data":"7e25c633944e5e419769ef918b04fcf8c871286b7ce6bd02b09dfd2170665395"}
Oct 30 22:08:14.566355 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:14.566274    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8k87n" podStartSLOduration=368.020127652 podStartE2EDuration="6m10.566258863s" totalImagesPullingTime="2.546131211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:14.565773593 +0000 UTC m=+34.774406267" watchObservedRunningTime="2026-10-30 22:08:14.566258863 +0000 UTC m=+34.774891536"
Oct 30 22:08:14.654365 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:14.654309    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:14.654516 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:14.654386    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:14.654516 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:14.654414    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:14.654607 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:14.654592    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:08:14.654663 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:14.654654    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:18.654636552 +0000 UTC m=+38.863269206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : secret "router-metrics-certs-default" not found
Oct 30 22:08:14.655071 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:14.655051    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:14.655152 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:14.655119    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls podName:557fc88c-2102-45ec-adc4-80bae4f140cb nodeName:}" failed. No retries permitted until 2026-10-30 22:08:18.655104806 +0000 UTC m=+38.863737462 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-k5qrx" (UID: "557fc88c-2102-45ec-adc4-80bae4f140cb") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:14.655217 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:14.655188    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:18.655176811 +0000 UTC m=+38.863809468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : configmap references non-existent config key: service-ca.crt
Oct 30 22:08:14.755737 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:14.755696    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:14.755904 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:14.755752    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:14.755904 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:14.755803    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:14.755904 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:14.755871    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:14.756090 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:14.755913    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:14.756090 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:14.756081    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:08:14.756196 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:14.756128    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert podName:ba52011d-6892-4b4d-ac14-07e8be859d9e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:18.756114439 +0000 UTC m=+38.964747104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8sk5t" (UID: "ba52011d-6892-4b4d-ac14-07e8be859d9e") : secret "networking-console-plugin-cert" not found
Oct 30 22:08:14.756196 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:14.756170    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:08:14.756348 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:14.756199    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls podName:a4369922-ceac-4aaf-9a8b-35c5a0f27d91 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:18.756188276 +0000 UTC m=+38.964820941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls") pod "dns-default-jg2dx" (UID: "a4369922-ceac-4aaf-9a8b-35c5a0f27d91") : secret "dns-default-metrics-tls" not found
Oct 30 22:08:14.756348 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:14.756256    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:14.756348 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:14.756265    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdbcb7997-ggr5g: secret "image-registry-tls" not found
Oct 30 22:08:14.756348 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:14.756293    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls podName:8969718f-1dae-4bde-81df-592ee58a1f76 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:18.756279287 +0000 UTC m=+38.964911951 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls") pod "image-registry-bdbcb7997-ggr5g" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76") : secret "image-registry-tls" not found
Oct 30 22:08:14.756575 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:14.756360    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:08:14.756575 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:14.756390    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls podName:610534ce-c054-41fd-8ab6-b70696b7e360 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:18.756380786 +0000 UTC m=+38.965013451 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-v6j52" (UID: "610534ce-c054-41fd-8ab6-b70696b7e360") : secret "samples-operator-tls" not found
Oct 30 22:08:14.756575 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:14.756435    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:14.756575 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:14.756451    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ff86975d4-6xkd8: secret "image-registry-tls" not found
Oct 30 22:08:14.756575 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:14.756483    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls podName:4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d nodeName:}" failed. No retries permitted until 2026-10-30 22:08:18.756472953 +0000 UTC m=+38.965105604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls") pod "image-registry-5ff86975d4-6xkd8" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d") : secret "image-registry-tls" not found
Oct 30 22:08:16.458525 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:16.458358    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-652qc" event={"ID":"a4254134-79fd-411a-a492-3e620d223ddf","Type":"ContainerStarted","Data":"412740f4f705faabbe68c8ea351e4d2f4f4d9c4a34f7bb30ca614a8a99438210"}
Oct 30 22:08:17.466051 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:17.466028    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/0.log"
Oct 30 22:08:17.466506 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:17.466070    2572 generic.go:356] "Generic (PLEG): container finished" podID="00c23fb6-cbfb-4498-ad17-15998e6d2d66" containerID="603f7fe4945bc76a40771d309714345c677a97f578e01cbf238bbc8623ab0952" exitCode=255
Oct 30 22:08:17.466506 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:17.466304    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr" event={"ID":"00c23fb6-cbfb-4498-ad17-15998e6d2d66","Type":"ContainerDied","Data":"603f7fe4945bc76a40771d309714345c677a97f578e01cbf238bbc8623ab0952"}
Oct 30 22:08:17.466851 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:17.466831    2572 scope.go:122] "RemoveContainer" containerID="603f7fe4945bc76a40771d309714345c677a97f578e01cbf238bbc8623ab0952"
Oct 30 22:08:17.587207 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:17.586780    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-652qc" podStartSLOduration=368.933734619 podStartE2EDuration="6m13.586760677s" totalImagesPullingTime="4.653026058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:16.587611296 +0000 UTC m=+36.796243980" watchObservedRunningTime="2026-10-30 22:08:17.586760677 +0000 UTC m=+37.795393351"
Oct 30 22:08:18.133770 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.133743    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dqpbw"]
Oct 30 22:08:18.162192 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.162158    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:18.249126 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.249097    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dqpbw"]
Oct 30 22:08:18.291510 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.291478    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:18.291675 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.291526    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdae7310-568d-48ec-80e7-841bd98ab04e-crio-socket\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:18.291675 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.291552    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:18.291675 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.291618    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdae7310-568d-48ec-80e7-841bd98ab04e-data-volume\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:18.291809 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.291693    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz4c8\" (UniqueName: \"kubernetes.io/projected/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-api-access-jz4c8\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:18.295070 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.295047    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 22:08:18.295070 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.295049    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nff88\""
Oct 30 22:08:18.296372 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.296351    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 22:08:18.392743 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.392215    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4c8\" (UniqueName: \"kubernetes.io/projected/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-api-access-jz4c8\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:18.392743 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.392325    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:18.392743 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.392503    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:08:18.392743 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.392565    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-rbac-proxy-cm podName:fdae7310-568d-48ec-80e7-841bd98ab04e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:18.892547894 +0000 UTC m=+39.101180548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dqpbw" (UID: "fdae7310-568d-48ec-80e7-841bd98ab04e") : configmap "kube-rbac-proxy" not found
Oct 30 22:08:18.393662 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.393120    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdae7310-568d-48ec-80e7-841bd98ab04e-crio-socket\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:18.393662 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.393166    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:18.393662 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.393246    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdae7310-568d-48ec-80e7-841bd98ab04e-data-volume\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:18.393662 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.393428    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdae7310-568d-48ec-80e7-841bd98ab04e-crio-socket\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:18.393662 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.393524    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:08:18.393662 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.393562    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls podName:fdae7310-568d-48ec-80e7-841bd98ab04e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:18.893549864 +0000 UTC m=+39.102182517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqpbw" (UID: "fdae7310-568d-48ec-80e7-841bd98ab04e") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:08:18.393662 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.393619    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdae7310-568d-48ec-80e7-841bd98ab04e-data-volume\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:18.460148 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.460114    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4c8\" (UniqueName: \"kubernetes.io/projected/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-api-access-jz4c8\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:18.473450 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.473417    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp" event={"ID":"09b48430-dd49-439d-b317-384408fec509","Type":"ContainerStarted","Data":"c19c7feede0fd2eb742fa41a05be6e9f0e9b53c3ee5c7eb4e8d97bdba4555e15"}
Oct 30 22:08:18.476106 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.476076    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-svwl7" event={"ID":"d81c8448-4580-4174-a7d8-28f5aadf238f","Type":"ContainerStarted","Data":"b40f3683e2a79d76a9215769fc3e92759d755c37b3238dea62874067b124e301"}
Oct 30 22:08:18.478706 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.478685    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/1.log"
Oct 30 22:08:18.479665 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.479639    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/0.log"
Oct 30 22:08:18.479762 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.479681    2572 generic.go:356] "Generic (PLEG): container finished" podID="00c23fb6-cbfb-4498-ad17-15998e6d2d66" containerID="bab56f6146fa081692f161b3f11e5251a5f21f023755d1e5896cb70b10996697" exitCode=255
Oct 30 22:08:18.479762 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.479754    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr" event={"ID":"00c23fb6-cbfb-4498-ad17-15998e6d2d66","Type":"ContainerDied","Data":"bab56f6146fa081692f161b3f11e5251a5f21f023755d1e5896cb70b10996697"}
Oct 30 22:08:18.480050 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.480035    2572 scope.go:122] "RemoveContainer" containerID="603f7fe4945bc76a40771d309714345c677a97f578e01cbf238bbc8623ab0952"
Oct 30 22:08:18.481157 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.480394    2572 scope.go:122] "RemoveContainer" containerID="bab56f6146fa081692f161b3f11e5251a5f21f023755d1e5896cb70b10996697"
Oct 30 22:08:18.484260 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.481250    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gs4xr_openshift-console-operator(00c23fb6-cbfb-4498-ad17-15998e6d2d66)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr" podUID="00c23fb6-cbfb-4498-ad17-15998e6d2d66"
Oct 30 22:08:18.594431 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.593423    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp" podStartSLOduration=369.161627401 podStartE2EDuration="6m14.593403358s" totalImagesPullingTime="5.431775957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:18.592298662 +0000 UTC m=+38.800931369" watchObservedRunningTime="2026-10-30 22:08:18.593403358 +0000 UTC m=+38.802036032"
Oct 30 22:08:18.696162 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.696130    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:18.696357 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.696174    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:18.696357 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.696192    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:18.696498 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.696477    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:08:18.696555 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.696544    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:26.696531523 +0000 UTC m=+46.905164173 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : secret "router-metrics-certs-default" not found
Oct 30 22:08:18.696892 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.696878    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:18.696969 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.696925    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls podName:557fc88c-2102-45ec-adc4-80bae4f140cb nodeName:}" failed. No retries permitted until 2026-10-30 22:08:26.696908876 +0000 UTC m=+46.905541532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-k5qrx" (UID: "557fc88c-2102-45ec-adc4-80bae4f140cb") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:18.697012 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.696999    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:26.69698653 +0000 UTC m=+46.905619182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : configmap references non-existent config key: service-ca.crt
Oct 30 22:08:18.711694 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.711380    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-svwl7" podStartSLOduration=2.740536886 podStartE2EDuration="8.711361042s" totalImagesPullingTime="5.970824156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:18.708631007 +0000 UTC m=+38.917263699" watchObservedRunningTime="2026-10-30 22:08:18.711361042 +0000 UTC m=+38.919993716"
Oct 30 22:08:18.796936 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.796900    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:18.797090 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.796963    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:18.797090 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.797046    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:18.797090 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.797071    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:18.797256 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.797119    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:18.797350 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.797310    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:08:18.797432 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.797402    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls podName:610534ce-c054-41fd-8ab6-b70696b7e360 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:26.797383774 +0000 UTC m=+47.006016441 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-v6j52" (UID: "610534ce-c054-41fd-8ab6-b70696b7e360") : secret "samples-operator-tls" not found
Oct 30 22:08:18.797506 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.797492    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:18.797560 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.797509    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ff86975d4-6xkd8: secret "image-registry-tls" not found
Oct 30 22:08:18.797560 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.797546    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls podName:4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d nodeName:}" failed. No retries permitted until 2026-10-30 22:08:26.797534854 +0000 UTC m=+47.006167511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls") pod "image-registry-5ff86975d4-6xkd8" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d") : secret "image-registry-tls" not found
Oct 30 22:08:18.797665 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.797608    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:08:18.797665 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.797643    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert podName:ba52011d-6892-4b4d-ac14-07e8be859d9e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:26.797630274 +0000 UTC m=+47.006262928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8sk5t" (UID: "ba52011d-6892-4b4d-ac14-07e8be859d9e") : secret "networking-console-plugin-cert" not found
Oct 30 22:08:18.797767 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.797697    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:08:18.797767 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.797727    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls podName:a4369922-ceac-4aaf-9a8b-35c5a0f27d91 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:26.797717184 +0000 UTC m=+47.006349837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls") pod "dns-default-jg2dx" (UID: "a4369922-ceac-4aaf-9a8b-35c5a0f27d91") : secret "dns-default-metrics-tls" not found
Oct 30 22:08:18.797868 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.797790    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:18.797868 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.797800    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdbcb7997-ggr5g: secret "image-registry-tls" not found
Oct 30 22:08:18.797868 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.797830    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls podName:8969718f-1dae-4bde-81df-592ee58a1f76 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:26.797820607 +0000 UTC m=+47.006453261 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls") pod "image-registry-bdbcb7997-ggr5g" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76") : secret "image-registry-tls" not found
Oct 30 22:08:18.897998 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.897786    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:18.897998 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:18.897831    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:18.897998 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.897900    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:08:18.897998 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.897957    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-rbac-proxy-cm podName:fdae7310-568d-48ec-80e7-841bd98ab04e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:19.897940051 +0000 UTC m=+40.106572723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dqpbw" (UID: "fdae7310-568d-48ec-80e7-841bd98ab04e") : configmap "kube-rbac-proxy" not found
Oct 30 22:08:18.898251 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.898016    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:08:18.898251 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:18.898091    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls podName:fdae7310-568d-48ec-80e7-841bd98ab04e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:19.898073491 +0000 UTC m=+40.106706156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqpbw" (UID: "fdae7310-568d-48ec-80e7-841bd98ab04e") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:08:19.130227 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.130150    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-642k8"]
Oct 30 22:08:19.148846 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.148809    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-642k8"
Oct 30 22:08:19.191636 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.191587    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-642k8"]
Oct 30 22:08:19.287176 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.287147    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 22:08:19.287633 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.287244    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 22:08:19.287633 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.287286    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dlplv\""
Oct 30 22:08:19.301557 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.301425    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n89g8\" (UniqueName: \"kubernetes.io/projected/76b48563-d38b-4e30-af57-c37cc7b66a9e-kube-api-access-n89g8\") pod \"migrator-66df5d496b-642k8\" (UID: \"76b48563-d38b-4e30-af57-c37cc7b66a9e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-642k8"
Oct 30 22:08:19.403395 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.402501    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n89g8\" (UniqueName: \"kubernetes.io/projected/76b48563-d38b-4e30-af57-c37cc7b66a9e-kube-api-access-n89g8\") pod \"migrator-66df5d496b-642k8\" (UID: \"76b48563-d38b-4e30-af57-c37cc7b66a9e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-642k8"
Oct 30 22:08:19.472748 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.472720    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n89g8\" (UniqueName: \"kubernetes.io/projected/76b48563-d38b-4e30-af57-c37cc7b66a9e-kube-api-access-n89g8\") pod \"migrator-66df5d496b-642k8\" (UID: \"76b48563-d38b-4e30-af57-c37cc7b66a9e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-642k8"
Oct 30 22:08:19.474473 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.474447    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-642k8"
Oct 30 22:08:19.485723 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.485693    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f8tgh" event={"ID":"7d098f8f-7073-4413-9add-d07ef7ce41e1","Type":"ContainerStarted","Data":"a57b4d19e911023ec241ab389f34caa193579983247dccf6daf57ac49024f94f"}
Oct 30 22:08:19.487801 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.487765    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/1.log"
Oct 30 22:08:19.488167 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.488148    2572 scope.go:122] "RemoveContainer" containerID="bab56f6146fa081692f161b3f11e5251a5f21f023755d1e5896cb70b10996697"
Oct 30 22:08:19.488389 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:19.488362    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gs4xr_openshift-console-operator(00c23fb6-cbfb-4498-ad17-15998e6d2d66)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr" podUID="00c23fb6-cbfb-4498-ad17-15998e6d2d66"
Oct 30 22:08:19.492521 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.490213    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7" event={"ID":"bb5baa5b-c887-42ea-b759-9347ae332051","Type":"ContainerStarted","Data":"a46b5a06f654ad29b8481476b2c6367c02e1249bd54bb655b102ad6c110e636c"}
Oct 30 22:08:19.602233 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.602182    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f8tgh" podStartSLOduration=365.131909396 podStartE2EDuration="6m12.602167256s" totalImagesPullingTime="7.47025786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:19.601497371 +0000 UTC m=+39.810130045" watchObservedRunningTime="2026-10-30 22:08:19.602167256 +0000 UTC m=+39.810799928"
Oct 30 22:08:19.838200 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.838137    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7" podStartSLOduration=368.644409405 podStartE2EDuration="6m15.83811561s" totalImagesPullingTime="7.193706205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:19.833957946 +0000 UTC m=+40.042590640" watchObservedRunningTime="2026-10-30 22:08:19.83811561 +0000 UTC m=+40.046748283"
Oct 30 22:08:19.869909 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.868601    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-642k8"]
Oct 30 22:08:19.872046 ip-10-0-133-218 kubenswrapper[2572]: W1030 22:08:19.871987    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b48563_d38b_4e30_af57_c37cc7b66a9e.slice/crio-3dd55d0f0e7c3edaecbfd56c26bd17cf2d17274eae69e3aa8d6092c185224e4d WatchSource:0}: Error finding container 3dd55d0f0e7c3edaecbfd56c26bd17cf2d17274eae69e3aa8d6092c185224e4d: Status 404 returned error can't find the container with id 3dd55d0f0e7c3edaecbfd56c26bd17cf2d17274eae69e3aa8d6092c185224e4d
Oct 30 22:08:19.908501 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.907644    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:19.908501 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:19.907696    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:19.908501 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:19.907986    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:08:19.908501 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:19.908051    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls podName:fdae7310-568d-48ec-80e7-841bd98ab04e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:21.908032097 +0000 UTC m=+42.116664773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqpbw" (UID: "fdae7310-568d-48ec-80e7-841bd98ab04e") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:08:19.908501 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:19.908098    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:08:19.908501 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:19.908129    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-rbac-proxy-cm podName:fdae7310-568d-48ec-80e7-841bd98ab04e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:21.908118667 +0000 UTC m=+42.116751336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dqpbw" (UID: "fdae7310-568d-48ec-80e7-841bd98ab04e") : configmap "kube-rbac-proxy" not found
Oct 30 22:08:20.494683 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:20.494644    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-642k8" event={"ID":"76b48563-d38b-4e30-af57-c37cc7b66a9e","Type":"ContainerStarted","Data":"3dd55d0f0e7c3edaecbfd56c26bd17cf2d17274eae69e3aa8d6092c185224e4d"}
Oct 30 22:08:20.496401 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:20.496366    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7h6kb" event={"ID":"7c1575a8-9288-4779-aab6-e5409b097e9f","Type":"ContainerStarted","Data":"75fe23cfd574a63cd020e4073d76b9280426b19902ab502ae60ad930b93a533b"}
Oct 30 22:08:20.612971 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:20.612901    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7h6kb" podStartSLOduration=33.554045178 podStartE2EDuration="39.612878878s" totalImagesPullingTime="6.0588337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:20.611761334 +0000 UTC m=+40.820394008" watchObservedRunningTime="2026-10-30 22:08:20.612878878 +0000 UTC m=+40.821511592"
Oct 30 22:08:21.305255 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:21.305215    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:21.305635 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:21.305609    2572 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:21.305748 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:21.305679    2572 scope.go:122] "RemoveContainer" containerID="bab56f6146fa081692f161b3f11e5251a5f21f023755d1e5896cb70b10996697"
Oct 30 22:08:21.305919 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:21.305897    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gs4xr_openshift-console-operator(00c23fb6-cbfb-4498-ad17-15998e6d2d66)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr" podUID="00c23fb6-cbfb-4498-ad17-15998e6d2d66"
Oct 30 22:08:21.499695 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:21.499670    2572 scope.go:122] "RemoveContainer" containerID="bab56f6146fa081692f161b3f11e5251a5f21f023755d1e5896cb70b10996697"
Oct 30 22:08:21.499695 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:21.499683    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:08:21.500070 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:21.499850    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gs4xr_openshift-console-operator(00c23fb6-cbfb-4498-ad17-15998e6d2d66)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr" podUID="00c23fb6-cbfb-4498-ad17-15998e6d2d66"
Oct 30 22:08:21.928512 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:21.928446    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:21.928512 ip-10-0-133-218 kubenswrapper[2572]: I1030 22:08:21.928487    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:21.928676 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:21.928571    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:08:21.928676 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:21.928640    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-rbac-proxy-cm podName:fdae7310-568d-48ec-80e7-841bd98ab04e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:25.928621525 +0000 UTC m=+46.137254196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dqpbw" (UID: "fdae7310-568d-48ec-80e7-841bd98ab04e") : configmap "kube-rbac-proxy" not found
Oct 30 22:08:21.928676 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:21.928666    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:08:21.928785 ip-10-0-133-218 kubenswrapper[2572]: E1030 22:08:21.928714    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls podName:fdae7310-568d-48ec-80e7-841bd98ab04e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:25.928701099 +0000 UTC m=+46.137333762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqpbw" (UID: "fdae7310-568d-48ec-80e7-841bd98ab04e") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:08:21.945037 ip-10-0-133-218 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 22:08:21.957756 ip-10-0-133-218 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 22:08:21.957922 ip-10-0-133-218 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 22:08:21.957969 ip-10-0-133-218 systemd[1]: kubelet.service: Consumed 3.029s CPU time.
Oct 30 22:08:21.969991 ip-10-0-133-218 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 22:08:22.064081 ip-10-0-133-218 kubenswrapper[4947]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:08:22.064081 ip-10-0-133-218 kubenswrapper[4947]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 22:08:22.064081 ip-10-0-133-218 kubenswrapper[4947]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:08:22.064081 ip-10-0-133-218 kubenswrapper[4947]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:08:22.066390 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066290    4947 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:08:22.066390 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066390    4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066394    4947 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066397    4947 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066400    4947 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066403    4947 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066406    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066412    4947 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066415    4947 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066419    4947 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066422    4947 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066424    4947 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066427    4947 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066430    4947 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066432    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066435    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066437    4947 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066440    4947 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066443    4947 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066445    4947 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066448    4947 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:08:22.066455 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066451    4947 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066453    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066456    4947 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066458    4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066462    4947 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066465    4947 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066468    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066471    4947 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066473    4947 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066477    4947 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066481    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066484    4947 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066487    4947 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066489    4947 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066492    4947 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066495    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066497    4947 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066500    4947 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066504    4947 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:08:22.066929 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066506    4947 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066509    4947 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066511    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066521    4947 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066525    4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066527    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066530    4947 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066533    4947 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066535    4947 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066538    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066540    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066543    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066546    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066549    4947 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066551    4947 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066554    4947 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066557    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066560    4947 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066562    4947 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066564    4947 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:08:22.067398 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066567    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066571    4947 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066575    4947 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066577    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066580    4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066582    4947 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066585    4947 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066587    4947 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066590    4947 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066592    4947 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066595    4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066597    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066601    4947 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066603    4947 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066606    4947 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066608    4947 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066610    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066613    4947 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066615    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:08:22.067867 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066618    4947 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066621    4947 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066623    4947 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066626    4947 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066629    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066631    4947 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066634    4947 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066636    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066638    4947 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066641    4947 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066643    4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066646    4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066648    4947 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066651    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066653    4947 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066656    4947 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066658    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066660    4947 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066663    4947 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:08:22.068321 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066665    4947 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066668    4947 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066670    4947 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066673    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066676    4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066678    4947 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066680    4947 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.066684    4947 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067433    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067441    4947 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067444    4947 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067447    4947 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067450    4947 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067453    4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067456    4947 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067458    4947 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067461    4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067465    4947 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067467    4947 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:08:22.068815 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067470    4947 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067472    4947 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067475    4947 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067478    4947 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067480    4947 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067484    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067486    4947 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067489    4947 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067491    4947 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067494    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067496    4947 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067499    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067501    4947 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067503    4947 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067506    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067508    4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067511    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067514    4947 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067517    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:08:22.069324 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067519    4947 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067522    4947 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067525    4947 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067528    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067530    4947 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067533    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067536    4947 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067540    4947 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067544    4947 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067546    4947 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067549    4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067552    4947 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067555    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067558    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067560    4947 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067563    4947 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067565    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067568    4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067570    4947 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:08:22.069783 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067573    4947 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067575    4947 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067578    4947 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067580    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067583    4947 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067585    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067588    4947 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067590    4947 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067592    4947 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067595    4947 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067597    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067600    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067603    4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067605    4947 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067607    4947 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067610    4947 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067613    4947 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067616    4947 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067619    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:08:22.070251 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067623    4947 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067627    4947 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067630    4947 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067632    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067635    4947 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067638    4947 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067641    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067643    4947 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067646    4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067652    4947 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067655    4947 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067658    4947 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067660    4947 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067662    4947 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067665    4947 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067667    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067670    4947 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067672    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067675    4947 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067677    4947 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:08:22.070723 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067680    4947 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067682    4947 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067684    4947 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067687    4947 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067690    4947 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067692    4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067694    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067697    4947 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067699    4947 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067702    4947 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067708    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067711    4947 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067714    4947 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067717    4947 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067719    4947 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067722    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067725    4947 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.067727    4947 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068237    4947 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068243    4947 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:08:22.071201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068247    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068249    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068252    4947 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068255    4947 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068257    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068260    4947 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068263    4947 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068265    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068268    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068273    4947 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068276    4947 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068279    4947 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068290    4947 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068293    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068295    4947 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068298    4947 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068300    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068303    4947 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068305    4947 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068308    4947 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:08:22.071690 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068311    4947 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068313    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068317    4947 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068320    4947 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068322    4947 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068325    4947 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068344    4947 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068348    4947 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068351    4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068353    4947 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068356    4947 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068358    4947 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068361    4947 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068363    4947 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068366    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068368    4947 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068371    4947 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068373    4947 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068376    4947 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068378    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:08:22.072201 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068381    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068385    4947 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068387    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068390    4947 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068392    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068395    4947 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068399    4947 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068403    4947 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068407    4947 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068410    4947 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068413    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068416    4947 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068419    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068422    4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068424    4947 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068428    4947 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068430    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068433    4947 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068436    4947 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:08:22.072729 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068438    4947 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068441    4947 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068443    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068446    4947 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068448    4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068451    4947 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068454    4947 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068457    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068459    4947 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068462    4947 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068464    4947 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068466    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068469    4947 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068472    4947 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068477    4947 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068479    4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068482    4947 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068484    4947 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068487    4947 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:08:22.073185 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068489    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068492    4947 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068494    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068497    4947 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068499    4947 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068502    4947 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068504    4947 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068506    4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068509    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068512    4947 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068515    4947 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068517    4947 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068520    4947 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068522    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068525    4947 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068527    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068529    4947 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068532    4947 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068534    4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068537    4947 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:08:22.073677 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068539    4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068542    4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068544    4947 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068547    4947 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068550    4947 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.068553    4947 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068651    4947 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068664    4947 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068672    4947 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068677    4947 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068682    4947 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068685    4947 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068689    4947 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068693    4947 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068696    4947 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068700    4947 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068703    4947 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068708    4947 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068711    4947 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068714    4947 flags.go:64] FLAG: --cgroup-root=""
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068716    4947 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068719    4947 flags.go:64] FLAG: --client-ca-file=""
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068722    4947 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 22:08:22.074163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068726    4947 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068731    4947 flags.go:64] FLAG: --cluster-domain=""
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068735    4947 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068738    4947 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068741    4947 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068744    4947 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068748    4947 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068751    4947 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068754    4947 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068758    4947 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068760    4947 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068764    4947 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068766    4947 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068770    4947 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068772    4947 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068776    4947 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068781    4947 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068784    4947 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068787    4947 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068790    4947 flags.go:64] FLAG: --enable-server="true"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068793    4947 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068797    4947 flags.go:64] FLAG: --event-burst="100"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068800    4947 flags.go:64] FLAG: --event-qps="50"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068803    4947 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 22:08:22.074871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068806    4947 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068809    4947 flags.go:64] FLAG: --eviction-hard=""
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068814    4947 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068817    4947 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068820    4947 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068823    4947 flags.go:64] FLAG: --eviction-soft=""
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068826    4947 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068829    4947 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068832    4947 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068836    4947 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068839    4947 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068842    4947 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068845    4947 flags.go:64] FLAG: --feature-gates=""
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068849    4947 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068852    4947 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068855    4947 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068858    4947 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068861    4947 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068864    4947 flags.go:64] FLAG: --help="false"
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068867    4947 flags.go:64] FLAG: --hostname-override="ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068870    4947 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068873    4947 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068876    4947 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 22:08:22.075845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068879    4947 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068884    4947 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068887    4947 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068890    4947 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068893    4947 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068895    4947 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068898    4947 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068901    4947 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068904    4947 flags.go:64] FLAG: --kube-reserved=""
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068907    4947 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068910    4947 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068914    4947 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068917    4947 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068920    4947 flags.go:64] FLAG: --lock-file=""
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068923    4947 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068925    4947 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068929    4947 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068934    4947 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068937    4947 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068941    4947 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068944    4947 flags.go:64] FLAG: --logging-format="text"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068946    4947 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068949    4947 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068952    4947 flags.go:64] FLAG: --manifest-url=""
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068955    4947 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 22:08:22.076524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068959    4947 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068962    4947 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068966    4947 flags.go:64] FLAG: --max-pods="110"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068969    4947 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068972    4947 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068975    4947 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068978    4947 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068981    4947 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068986    4947 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068988    4947 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068995    4947 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.068999    4947 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069001    4947 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069004    4947 flags.go:64] FLAG: --pod-cidr=""
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069007    4947 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069010    4947 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069013    4947 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069016    4947 flags.go:64] FLAG: --port="10250"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069020    4947 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069023    4947 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0b3d36b757523b2ee"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069027    4947 flags.go:64] FLAG: --qos-reserved=""
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069029    4947 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069033    4947 flags.go:64] FLAG: --register-node="true"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069035    4947 flags.go:64] FLAG: --register-with-taints=""
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069039    4947 flags.go:64] FLAG: --registry-burst="10"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069042    4947 flags.go:64] FLAG: --registry-qps="5"
Oct 30 22:08:22.077123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069045    4947 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069072    4947 flags.go:64] FLAG: --reserved-memory=""
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069083    4947 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069087    4947 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069090    4947 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069093    4947 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069095    4947 flags.go:64] FLAG: --runonce="false"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069098    4947 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069101    4947 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069104    4947 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069107    4947 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069109    4947 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069112    4947 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069116    4947 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069120    4947 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069123    4947 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069126    4947 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069129    4947 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069132    4947 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069135    4947 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069137    4947 flags.go:64] FLAG: --system-cgroups=""
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069140    4947 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069146    4947 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069149    4947 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069152    4947 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 22:08:22.077746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069160    4947 flags.go:64] FLAG: --tls-min-version=""
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069163    4947 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069166    4947 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069169    4947 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069172    4947 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069175    4947 flags.go:64] FLAG: --v="2"
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069180    4947 flags.go:64] FLAG: --version="false"
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069184    4947 flags.go:64] FLAG: --vmodule=""
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069188    4947 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069191    4947 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069356    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069361    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069364    4947 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069367    4947 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069370    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069373    4947 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069375    4947 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069378    4947 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069380    4947 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069383    4947 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069385    4947 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069389    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:08:22.078509 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069393    4947 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069395    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069398    4947 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069401    4947 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069403    4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069406    4947 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069408    4947 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069411    4947 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069413    4947 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069417    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069420    4947 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069423    4947 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069425    4947 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069428    4947 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069430    4947 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069433    4947 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069435    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069438    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069440    4947 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069443    4947 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:08:22.079044 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069446    4947 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069449    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069451    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069453    4947 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069456    4947 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069459    4947 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069461    4947 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069464    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069466    4947 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069469    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069471    4947 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069475    4947 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069478    4947 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069482    4947 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069485    4947 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069489    4947 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069492    4947 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069494    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069497    4947 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:08:22.079555 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069500    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069502    4947 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069506    4947 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069508    4947 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069511    4947 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069514    4947 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069516    4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069519    4947 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069521    4947 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069524    4947 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069527    4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069529    4947 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069532    4947 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069534    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069537    4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069540    4947 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069542    4947 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069545    4947 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069547    4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069550    4947 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:08:22.080004 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069552    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069555    4947 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069557    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069560    4947 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069563    4947 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069566    4947 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069568    4947 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069571    4947 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069573    4947 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069576    4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069578    4947 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069581    4947 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069583    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069587    4947 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069592    4947 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069595    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069598    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069601    4947 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069603    4947 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:08:22.080590 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069606    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:08:22.081047 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069609    4947 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:08:22.081047 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069611    4947 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:08:22.081047 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069614    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:08:22.081047 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069617    4947 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:08:22.081047 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069619    4947 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:08:22.081047 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069622    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:08:22.081047 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069625    4947 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:08:22.081047 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069628    4947 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:08:22.081047 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069630    4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:08:22.081047 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069633    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:08:22.081047 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069635    4947 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:08:22.081047 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069638    4947 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:08:22.081047 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069640    4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:08:22.081047 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069643    4947 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:08:22.081047 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069645    4947 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:08:22.081047 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.069672    4947 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069717    4947 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069722    4947 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069725    4947 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069728    4947 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069730    4947 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069733    4947 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069736    4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069738    4947 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069741    4947 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069743    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069747    4947 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069753    4947 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069756    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069758    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069761    4947 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069764    4947 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069766    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069769    4947 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:08:22.081456 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069771    4947 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069774    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069777    4947 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069779    4947 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069782    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069784    4947 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069793    4947 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069796    4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069798    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069801    4947 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069803    4947 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069806    4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069808    4947 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069810    4947 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069815    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069817    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069820    4947 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069822    4947 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069825    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:08:22.081967 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069827    4947 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069830    4947 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069832    4947 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069835    4947 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069837    4947 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069840    4947 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069843    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069846    4947 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069848    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069851    4947 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069854    4947 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069856    4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069858    4947 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069861    4947 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069864    4947 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069866    4947 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069869    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069871    4947 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069874    4947 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069876    4947 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:08:22.082504 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069884    4947 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069893    4947 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069896    4947 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069898    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069901    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069904    4947 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069906    4947 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069910    4947 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069913    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069915    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069918    4947 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069920    4947 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069923    4947 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069925    4947 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069928    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069930    4947 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069933    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069935    4947 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069940    4947 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:08:22.083110 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069942    4947 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069946    4947 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069950    4947 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069953    4947 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069956    4947 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069959    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069962    4947 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069965    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069967    4947 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069970    4947 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069972    4947 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069975    4947 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069977    4947 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069980    4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069988    4947 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069991    4947 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069993    4947 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069996    4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.069998    4947 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:08:22.083688 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.070001    4947 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.070005    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.070007    4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.070010    4947 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.070012    4947 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.070015    4947 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.070018    4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.070020    4947 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.070023    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.070025    4947 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.070028    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.073980    4947 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.073995    4947 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.074012    4947 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.074016    4947 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074073    4947 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074078    4947 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074082    4947 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074085    4947 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074088    4947 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:08:22.084166 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074090    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074093    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074096    4947 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074099    4947 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074101    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074104    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074107    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074112    4947 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074115    4947 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074118    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074120    4947 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074123    4947 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074125    4947 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074128    4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074130    4947 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074133    4947 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074135    4947 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074138    4947 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074140    4947 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074143    4947 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:08:22.084683 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074145    4947 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074148    4947 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074151    4947 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074153    4947 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074156    4947 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074158    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074161    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074164    4947 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074168    4947 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074171    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074174    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074178    4947 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074181    4947 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074185    4947 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074189    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074192    4947 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074195    4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074198    4947 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:08:22.085164 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074201    4947 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074204    4947 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074208    4947 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074211    4947 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074214    4947 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074216    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074218    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074221    4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074223    4947 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074226    4947 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074228    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074231    4947 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074233    4947 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074236    4947 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074238    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074241    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074244    4947 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074247    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074249    4947 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074252    4947 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:08:22.085610 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074255    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074257    4947 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074259    4947 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074262    4947 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074264    4947 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074267    4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074269    4947 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074272    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074275    4947 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074277    4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074280    4947 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074282    4947 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074284    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074287    4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074289    4947 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074292    4947 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074294    4947 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074297    4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074299    4947 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:08:22.086183 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074302    4947 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074304    4947 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074307    4947 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074309    4947 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074312    4947 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074314    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074317    4947 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074320    4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074322    4947 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074325    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074327    4947 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074351    4947 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074354    4947 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074356    4947 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074359    4947 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074361    4947 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074364    4947 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074367    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074369    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074372    4947 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:08:22.086657 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074375    4947 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:08:22.087130 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074377    4947 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:08:22.087130 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074380    4947 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:08:22.087130 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074382    4947 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:08:22.087130 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.074411    4947 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:08:22.087130 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.074790    4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:08:22.087130 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075209    4947 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:08:22.087130 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075216    4947 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:08:22.087130 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075229    4947 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:08:22.087130 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075235    4947 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:08:22.087130 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075239    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:08:22.087130 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075244    4947 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:08:22.087130 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075248    4947 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:08:22.087130 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075253    4947 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:08:22.087130 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075257    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:08:22.087130 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075261    4947 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:08:22.087130 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075265    4947 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075271    4947 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075275    4947 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075279    4947 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075283    4947 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075293    4947 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075297    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075302    4947 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075306    4947 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075310    4947 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075314    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075318    4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075322    4947 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075326    4947 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075397    4947 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075403    4947 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075408    4947 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075413    4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075423    4947 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075442    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:08:22.087570 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075446    4947 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075451    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075456    4947 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075460    4947 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075464    4947 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075468    4947 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075473    4947 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075478    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075482    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075486    4947 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075491    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075500    4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075504    4947 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075508    4947 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075512    4947 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075516    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075521    4947 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075525    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075529    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075533    4947 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:08:22.088060 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075538    4947 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075542    4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075546    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075556    4947 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075561    4947 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075565    4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075572    4947 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075578    4947 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075583    4947 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075587    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075592    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075596    4947 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075600    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075605    4947 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075610    4947 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075619    4947 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075623    4947 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075628    4947 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075632    4947 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:08:22.088564 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075637    4947 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075641    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075645    4947 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075649    4947 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075653    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075657    4947 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075661    4947 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075665    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075669    4947 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075678    4947 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075682    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075686    4947 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075691    4947 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075695    4947 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075699    4947 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075703    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075709    4947 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075716    4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075721    4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:08:22.089017 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075726    4947 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:08:22.089534 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075730    4947 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:08:22.089534 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075739    4947 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:08:22.089534 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075743    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:08:22.089534 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075747    4947 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:08:22.089534 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075751    4947 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:08:22.089534 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075755    4947 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:08:22.089534 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075759    4947 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:08:22.089534 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075763    4947 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:08:22.089534 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075767    4947 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:08:22.089534 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075771    4947 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:08:22.089534 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075775    4947 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:08:22.089534 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075779    4947 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:08:22.089534 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075783    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:08:22.089534 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075788    4947 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:08:22.089534 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075798    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:08:22.089534 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.075802    4947 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:08:22.089931 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.075860    4947 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:08:22.089931 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076034    4947 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:08:22.089931 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076133    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:08:22.089931 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076140    4947 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:08:22.089931 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076145    4947 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:08:22.089931 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076151    4947 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:08:22.089931 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076155    4947 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:08:22.089931 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076159    4947 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:08:22.089931 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076164    4947 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:08:22.089931 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076168    4947 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:08:22.089931 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076172    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:08:22.089931 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076186    4947 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:08:22.089931 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076191    4947 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:08:22.089931 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076195    4947 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:08:22.089931 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076199    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:08:22.089931 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076203    4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076207    4947 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076211    4947 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076215    4947 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076220    4947 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076224    4947 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076228    4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076233    4947 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076238    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076242    4947 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076246    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076250    4947 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076257    4947 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076261    4947 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076265    4947 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076268    4947 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076273    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076277    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076288    4947 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076294    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:08:22.090311 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076298    4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076302    4947 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076306    4947 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076310    4947 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076314    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076318    4947 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076322    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076326    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076353    4947 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076358    4947 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076362    4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076366    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076371    4947 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076376    4947 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076381    4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076387    4947 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076393    4947 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076398    4947 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076401    4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:08:22.090811 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076406    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076413    4947 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076417    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076424    4947 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076428    4947 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076432    4947 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076436    4947 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076441    4947 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076443    4947 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076446    4947 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076449    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076452    4947 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076456    4947 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076459    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076461    4947 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076464    4947 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076467    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076469    4947 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076471    4947 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076474    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:08:22.091299 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076477    4947 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076479    4947 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076482    4947 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076484    4947 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076489    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076493    4947 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076498    4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076501    4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076504    4947 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076507    4947 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076509    4947 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076512    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076515    4947 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076517    4947 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076520    4947 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076523    4947 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076526    4947 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076528    4947 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076531    4947 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:08:22.091871 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076534    4947 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076537    4947 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076539    4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076542    4947 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076545    4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076548    4947 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076551    4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076553    4947 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076556    4947 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076560    4947 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076562    4947 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076565    4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076567    4947 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:22.076570    4947 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.076638    4947 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.077638    4947 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.077695    4947 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.077882    4947 server.go:1014] "Starting client certificate rotation"
Oct 30 22:08:22.092549 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.078014    4947 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:08:22.093320 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.078057    4947 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 22:02:40 +0000 UTC" deadline="2028-04-01 01:36:27.798334407 +0000 UTC"
Oct 30 22:08:22.093320 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.078085    4947 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12435h28m5.720251603s"
Oct 30 22:08:22.093320 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.078400    4947 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:08:22.093320 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.079488    4947 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:08:22.093320 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.081123    4947 log.go:25] "Validated CRI v1 runtime API"
Oct 30 22:08:22.093320 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.082771    4947 log.go:25] "Validated CRI v1 image API"
Oct 30 22:08:22.093320 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.083551    4947 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 22:08:22.093320 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.085475    4947 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8e076779-6cd5-400d-b4ca-994c866b4b29:/dev/nvme0n1p4 e42f82d1-72e2-4427-8c5a-3cd580b53f1b:/dev/nvme0n1p3]
Oct 30 22:08:22.093575 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.085489    4947 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1502caa923a2b2ab695fb6b2b2f7ef5d1b4e3b7dc87fba00ba473f880ead9d4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1502caa923a2b2ab695fb6b2b2f7ef5d1b4e3b7dc87fba00ba473f880ead9d4a/userdata/shm major:0 minor:252 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e059cdea8af69c2f35c6f2c86433cdfc163872947514209381afd25293ce48d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e059cdea8af69c2f35c6f2c86433cdfc163872947514209381afd25293ce48d/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f1cdb7881fbb080453f7bb6c53fbac80537ed17128b473b464984c2a6e4db5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f1cdb7881fbb080453f7bb6c53fbac80537ed17128b473b464984c2a6e4db5c/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f94e750fc006efb749803797e006255ddf0491f5f89e99505b03a099ef8f8b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f94e750fc006efb749803797e006255ddf0491f5f89e99505b03a099ef8f8b7/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33577b627b4a1735308d81e8e822ca8e9bbf7a1d1c2f3477dc5c6d6aa34ca360/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33577b627b4a1735308d81e8e822ca8e9bbf7a1d1c2f3477dc5c6d6aa34ca360/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33a5dce8aae11c140624982e5f02ee86456363d0feae26297084558448dd7501/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33a5dce8aae11c140624982e5f02ee86456363d0feae26297084558448dd7501/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34feb426af22aebf71fd4d331f0f8b048ce191d95ae6ee3c1ae6cc991c4a706a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34feb426af22aebf71fd4d331f0f8b048ce191d95ae6ee3c1ae6cc991c4a706a/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35f92cea187b4b04177b4d09b48957700237301a7d7abafbb402a40cd1305055/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35f92cea187b4b04177b4d09b48957700237301a7d7abafbb402a40cd1305055/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b61d4362c209461fe92640752244e15801e90a93e0168e7597b947a56a628c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b61d4362c209461fe92640752244e15801e90a93e0168e7597b947a56a628c7/userdata/shm major:0 minor:52 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3dd55d0f0e7c3edaecbfd56c26bd17cf2d17274eae69e3aa8d6092c185224e4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3dd55d0f0e7c3edaecbfd56c26bd17cf2d17274eae69e3aa8d6092c185224e4d/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4bf351d84cf28fc3d5c700f4d9c5dbe3275988880a55fa29234c6e3e1b99d073/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4bf351d84cf28fc3d5c700f4d9c5dbe3275988880a55fa29234c6e3e1b99d073/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/619bd399d0079020184e9e977bac7ce21130ffd14b03696dd52ce4a212f73aa0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/619bd399d0079020184e9e977bac7ce21130ffd14b03696dd52ce4a212f73aa0/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6dd646d0a376f2b2fcfcdb457563de9213574005035152738337b593023b105c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6dd646d0a376f2b2fcfcdb457563de9213574005035152738337b593023b105c/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d137caaea3f768787aeb03e28ac7f21294d6e608030e0c363d6939b4c46519f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d137caaea3f768787aeb03e28ac7f21294d6e608030e0c363d6939b4c46519f/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ed222783bdd148dbf9df772ddd6dae9cb919c817293fa2dc74f531b1e2c093a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ed222783bdd148dbf9df772ddd6dae9cb919c817293fa2dc74f531b1e2c093a/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91f886b0105538a43484d74538321d69e1c80487ecb28ab205584d477e690228/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91f886b0105538a43484d74538321d69e1c80487ecb28ab205584d477e690228/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/931bbe7eba22c56243937d2b9926956457d6ca1dc097a15c502a24c087717a23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/931bbe7eba22c56243937d2b9926956457d6ca1dc097a15c502a24c087717a23/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93cebfd8e1701b6f66ce8b91ee9f0e4ad453336d4fdee5ac05f97d618e9607f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93cebfd8e1701b6f66ce8b91ee9f0e4ad453336d4fdee5ac05f97d618e9607f1/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/955876a99ad96b940d6504588c872a5ee075b62107954327eb2b6b1f5a086035/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/955876a99ad96b940d6504588c872a5ee075b62107954327eb2b6b1f5a086035/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0fe66d1f732276bd45a6520e5b59a121b279c81db215d74ca0ef78832325c8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0fe66d1f732276bd45a6520e5b59a121b279c81db215d74ca0ef78832325c8b/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0eada557aeb34035fd734d4b685481c39bebcff32b33ee74351b9513eb3e22a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0eada557aeb34035fd734d4b685481c39bebcff32b33ee74351b9513eb3e22a/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc134333d3bbaed5e72f2b58f271684b858ac448fa30ce00164b295bfae297f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc134333d3bbaed5e72f2b58f271684b858ac448fa30ce00164b295bfae297f3/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f41d8d95231e5c3bf2162b438bdae00456b0e9b7d93677891f6adff38147e21c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f41d8d95231e5c3bf2162b438bdae00456b0e9b7d93677891f6adff38147e21c/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00c23fb6-cbfb-4498-ad17-15998e6d2d66/volumes/kubernetes.io~projected/kube-api-access-b8nn2:{mountpoint:/var/lib/kubelet/pods/00c23fb6-cbfb-4498-ad17-15998e6d2d66/volumes/kubernetes.io~projected/kube-api-access-b8nn2 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00c23fb6-cbfb-4498-ad17-15998e6d2d66/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/00c23fb6-cbfb-4498-ad17-15998e6d2d66/volumes/kubernetes.io~secret/serving-cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09b48430-dd49-439d-b317-384408fec509/volumes/kubernetes.io~projected/kube-api-access-f8kpl:{mountpoint:/var/lib/kubelet/pods/09b48430-dd49-439d-b317-384408fec509/volumes/kubernetes.io~projected/kube-api-access-f8kpl major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09b48430-dd49-439d-b317-384408fec509/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/09b48430-dd49-439d-b317-384408fec509/volumes/kubernetes.io~secret/serving-cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2188065c-aa72-45ea-a0bb-9665fe8f51a2/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2188065c-aa72-45ea-a0bb-9665fe8f51a2/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3788135a-d420-4738-a292-845c50e486a7/volumes/kubernetes.io~projected/kube-api-access-6c4bd:{mountpoint:/var/lib/kubelet/pods/3788135a-d420-4738-a292-845c50e486a7/volumes/kubernetes.io~projected/kube-api-access-6c4bd major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41dcf151-1d72-4978-b0aa-2046f213cb7e/volumes/kubernetes.io~projected/kube-api-access-rgr7f:{mountpoint:/var/lib/kubelet/pods/41dcf151-1d72-4978-b0aa-2046f213cb7e/volumes/kubernetes.io~projected/kube-api-access-rgr7f major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d/volumes/kubernetes.io~projected/kube-api-access-fx7tp:{mountpoint:/var/lib/kubelet/pods/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d/volumes/kubernetes.io~projected/kube-api-access-fx7tp major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/557fc88c-2102-45ec-adc4-80bae4f140cb/volumes/kubernetes.io~projected/kube-api-access-h4pmg:{mountpoint:/var/lib/kubelet/pods/557fc88c-2102-45ec-adc4-80bae4f140cb/volumes/kubernetes.io~projected/kube-api-access-h4pmg major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/610534ce-c054-41fd-8ab6-b70696b7e360/volumes/kubernetes.io~projected/kube-api-access-v7n8j:{mountpoint:/var/lib/kubelet/pods/610534ce-c054-41fd-8ab6-b70696b7e360/volumes/kubernetes.io~projected/kube-api-access-v7n8j major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7048b86e-5ca2-46c8-8b2d-79116829faf2/volumes/kubernetes.io~projected/kube-api-access-p4s4h:{mountpoint:/var/lib/kubelet/pods/7048b86e-5ca2-46c8-8b2d-79116829faf2/volumes/kubernetes.io~projected/kube-api-access-p4s4h major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76b48563-d38b-4e30-af57-c37cc7b66a9e/volumes/kubernetes.io~projected/kube-api-access-n89g8:{mountpoint:/var/lib/kubelet/pods/76b48563-d38b-4e30-af57-c37cc7b66a9e/volumes/kubernetes.io~projected/kube-api-access-n89g8 major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79823d82-b501-47c9-acae-667e192d721b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/79823d82-b501-47c9-acae-667e192d721b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79823d82-b501-47c9-acae-667e192d721b/volumes/kubernetes.io~projected/kube-api-access-gkbv8:{mountpoint:/var/lib/kubelet/pods/79823d82-b501-47c9-acae-667e192d721b/volumes/kubernetes.io~projected/kube-api-access-gkbv8 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79823d82-b501-47c9-acae-667e192d721b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/79823d82-b501-47c9-acae-667e192d721b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c1575a8-9288-4779-aab6-e5409b097e9f/volumes/kubernetes.io~projected/kube-api-access-nskz5:{mountpoint:/var/lib/kubelet/pods/7c1575a8-9288-4779-aab6-e5409b097e9f/volumes/kubernetes.io~projected/kube-api-access-nskz5 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d098f8f-7073-4413-9add-d07ef7ce41e1/volumes/kubernetes.io~projected/kube-api-access-trn7v:{mountpoint:/var/lib/kubelet/pods/7d098f8f-7073-4413-9add-d07ef7ce41e1/volumes/kubernetes.io~projected/kube-api-access-trn7v major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8969718f-1dae-4bde-81df-592ee58a1f76/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8969718f-1dae-4bde-81df-592ee58a1f76/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8969718f-1dae-4bde-81df-592ee58a1f76/volumes/kubernetes.io~projected/kube-api-access-rwmpt:{mountpoint:/var/lib/kubelet/pods/8969718f-1dae-4bde-81df-592ee58a1f76/volumes/kubernetes.io~projected/kube-api-access-rwmpt major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8969718f-1dae-4bde-81df-592ee58a1f76/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8969718f-1dae-4bde-81df-592ee58a1f76/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8969718f-1dae-4bde-81df-592ee58a1f76/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8969718f-1dae-4bde-81df-592ee58a1f76/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93f981a6-4af6-4405-aec6-89e539c42c67/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/93f981a6-4af6-4405-aec6-89e539c42c67/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94490976-1f2d-4db5-980a-322cad7c057b/volumes/kubernetes.io~projected/kube-api-access-vppkm:{mountpoint:/var/lib/kubelet/pods/94490976-1f2d-4db5-980a-322cad7c057b/volumes/kubernetes.io~projected/kube-api-access-vppkm major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94490976-1f2d-4db5-980a-322cad7c057b/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/94490976-1f2d-4db5-980a-322cad7c057b/volumes/kubernetes.io~secret/default-certificate major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94490976-1f2d-4db5-980a-322cad7c057b/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/94490976-1f2d-4db5-980a-322cad7c057b/volumes/kubernetes.io~secret/stats-auth major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4254134-79fd-411a-a492-3e620d223ddf/volumes/kubernetes.io~projected/kube-api-access-l5cv2:{mountpoint:/var/lib/kubelet/pods/a4254134-79fd-411a-a492-3e620d223ddf/volumes/kubernetes.io~projected/kube-api-access-l5cv2 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4254134-79fd-411a-a492-3e620d223ddf/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a4254134-79fd-411a-a492-3e620d223ddf/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4369922-ceac-4aaf-9a8b-35c5a0f27d91/volumes/kubernetes.io~projected/kube-api-access-jl8s9:{mountpoint:/var/lib/kubelet/pods/a4369922-ceac-4aaf-9a8b-35c5a0f27d91/volumes/kubernetes.io~projected/kube-api-access-jl8s9 major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd/volumes/kubernetes.io~projected/kube-api-access-j2q5s:{mountpoint:/var/lib/kubelet/pods/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd/volumes/kubernetes.io~projected/kube-api-access-j2q5s major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4d3f83c-7808-4a9a-84d1-d9428d5bd592/volumes/kubernetes.io~projected/kube-api-access-p4ww6:{mountpoint:/var/lib/kubelet/pods/b4d3f83c-7808-4a9a-84d1-d9428d5bd592/volumes/kubernetes.io~projected/kube-api-access-p4ww6 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9d3d620-4040-45ca-86da-ce6ba161ba3a/volumes/kubernetes.io~projected/kube-api-access-j7s7n:{mountpoint:/var/lib/kubelet/pods/b9d3d620-4040-45ca-86da-ce6ba161ba3a/volumes/kubernetes.io~projected/kube-api-access-j7s7n major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba0fc359-3a19-49ea-b798-2ca0080b30b9/volumes/kubernetes.io~projected/kube-api-access-8mkwp:{mountpoint:/var/lib/kubelet/pods/ba0fc359-3a19-49ea-b798-2ca0080b30b9/volumes/kubernetes.io~projected/kube-api-access-8mkwp major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb5baa5b-c887-42ea-b759-9347ae332051/volumes/kubernetes.io~projected/kube-api-access-9vss5:{mountpoint:/var/lib/kubelet/pods/bb5baa5b-c887-42ea-b759-9347ae332051/volumes/kubernetes.io~projected/kube-api-access-9vss5 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb5baa5b-c887-42ea-b759-9347ae332051/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/bb5baa5b-c887-42ea-b759-9347ae332051/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bec12d9c-76a5-463a-991e-ca82c18fccd2/volumes/kubernetes.io~projected/kube-api-access-jgn2d:{mountpoint:/var/lib/kubelet/pods/bec12d9c-76a5-463a-991e-ca82c18fccd2/volumes/kubernetes.io~projected/kube-api-access-jgn2d major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a30486-a1f5-4404-8303-6efed2dad370/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c0a30486-a1f5-4404-8303-6efed2dad370/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a30486-a1f5-4404-8303-6efed2dad370/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c0a30486-a1f5-4404-8303-6efed2dad370/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a30486-a1f5-4404-8303-6efed2dad370/volumes/kubernetes.io~projected/kube-api-access-dsdc4:{mountpoint:/var/lib/kubelet/pods/c0a30486-a1f5-4404-8303-6efed2dad370/volumes/kubernetes.io~projected/kube-api-access-dsdc4 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d81c8448-4580-4174-a7d8-28f5aadf238f/volumes/kubernetes.io~projected/kube-api-access-vkwmz:{mountpoint:/var/lib/kubelet/pods/d81c8448-4580-4174-a7d8-28f5aadf238f/volumes/kubernetes.io~projected/kube-api-access-vkwmz major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d81c8448-4580-4174-a7d8-28f5aadf238f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/d81c8448-4580-4174-a7d8-28f5aadf238f/volumes/kubernetes.io~secret/cert major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3555d3-b636-4047-957a-632dd893372e/volumes/kubernetes.io~projected/kube-api-access-64d4l:{mountpoint:/var/lib/kubelet/pods/da3555d3-b636-4047-957a-632dd893372e/volumes/kubernetes.io~projected/kube-api-access-64d4l major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f92ebb16-2530-4f59-98d6-c3c2695ad9ba/volumes/kubernetes.io~projected/kube-api-access-xllgn:{mountpoint:/var/lib/kubelet/pods/f92ebb16-2530-4f59-98d6-c3c2695ad9ba/volumes/kubernetes.io~projected/kube-api-access-xllgn major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcb4abba-e59f-4e64-af78-f1e60697b4b2/volumes/kubernetes.io~projected/kube-api-access-mtpfd:{mountpoint:/var/lib/kubelet/pods/fcb4abba-e59f-4e64-af78-f1e60697b4b2/volumes/kubernetes.io~projected/kube-api-access-mtpfd major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdae7310-568d-48ec-80e7-841bd98ab04e/volumes/kubernetes.io~projected/kube-api-access-jz4c8:{mountpoint:/var/lib/kubelet/pods/fdae7310-568d-48ec-80e7-841bd98ab04e/volumes/kubernetes.io~projected/kube-api-access-jz4c8 major:0 minor:316 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/0d14ab7ea5ae402141439e6129074d3dad553e521b910f53790b28b07d6aab72/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/3127f6dff07d4f5faea412fc300422895a44ca9ea643c9dbce45c715d2bb5a3e/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/29a5bde9d941aa0955662382e9909b42ad73e649c9e13e172521a8e2021002d5/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/1558e9d7b911bf96fcebd3c0eef1d503159ab9d772a530d90484da5fd590c0ee/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7b990ee2f22302c8ee905b04c5eda8968d35b40408966a8b6574b8d23efe623a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/9ec831585fdc86d7b885fcf03eb72213f454b5b187cc6f8e582f05c6b69cc123/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/df2bdfdd4697edda5831fd9c8f81b1e9139f403379c11c0ee99f1f2cdc560693/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/90989825fe20b1d989dd0387d7c7f9584cfe5fa4c7640d1fd3ce3d5a89adb0f9/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/81527b8bc1717b8621f247c8e0c4e72b4f79d1f0441551f790e3ba34daf2c989/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/2f456a0f2e8c3ba4348475bea8e509c40e0d3fa0a97b8136c963c3bfd0dbdfcf/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/6a1369feb49a98117cdbc2c23ffdc76e36b50b25f16a125766c98f7818c44d64/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/c8315cfc9a0a997e02d238f9be466bb61298570d8b690039d429a89f6b639647/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/2fb5408955e5d112dfa82a713a6ee1d9d52fed58df2c802fe6f53a485186cd4c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-153:{mountpoint:/var/lib/containers/storage/overlay/4a2d8f66182b5dc21a7e77f0fa2c78a1cf627ca3906b8eb3d8cb31fb7b678941/merged major:0 minor:153 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/ff1c8c0b0c44778a502b7da0843402a75a633289d525a8885d3bdfb578135350/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0e99ce121387710fd2fbb724cecca7883253546dc051d0477724d32206f967a6/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4a65cab21596a355755a9c50fd6bad3149948f49f88af65a34c60670153e0a08/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d263cc92d745660487aa4c23c1b91515f2cdc9c7e6a6d3d9a77e9a974f378790/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/792265592d18c89050fd51b9b18d61ff72bd769b5f3a991446b84a9241d54ffc/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f8d3e416746a190cb499445d88ae5e21b5fa31e3ec87d1092b958f5c15dd8b45/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8a0c4fc080a8fbbeaa80287fc01513f1bc7e0f9191baa2a403e96d90619000d5/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/000956c659f9fe14bd8ef511f75c31ead389958d9673c51f6c418095e8725263/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/7d57f854be217032de3bf7fd7ee5b5aace7651ef2a86543943f14a4da1630d67/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/337ea5464cb922819deef1b9173254856416aea48d98dc29a12b431dce6e221b/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/c67bb81664d014474e6aa0328f99b272286f7b8050ba64ccdd8539a4eb22dbfb/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/f983169a21114c95d7f4e4bb3f705227bd493690c3ec6e4ca2b027ab95093261/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/c452bee716a8e32d496e8b065428b65579a7e4cd5b20e832a6de7c4d7f6c6973/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/0c6e5a6969014d35ee9dad47dab1c8c9436796f14b1eeb58a2e8ad3ac8280358/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/1c4776b1981fd635ac6283edc1b43e6efc7fc5daa45b365c90984789b3952691/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/94bc7186b0ddeb2fbdc149520752ba093a9c34a708810eed8ba498d1fe3395a2/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/5e67ace219471470498dde905fd4d14c0119e69d9469d30e5dca25ec5854f2a8/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/9ca68a68a229b26b791e462ea48759186149f1e5121651db92f49f7ffe84c62a/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/6cedb175f9040ecf683d0ec44fff2ffab74e2038391a80d39df8f0bfa50499ec/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/104ee9252838c7f6eaf90754ba4b65d9e0b1804f6630e99521bede26646dae09/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/331ac58a6d9abcc0d8e3c4ce4c7ed6ea8e452c53d0ee4be7433e237b1c455fb6/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/e0e508d11fb2fd0d2d1b5cfc87b9d1c6c170a8c88d484429c257f3680dae83db/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/543ab037d069057ad4c153974f0f394d3c6ec915e61c10235835ad91e3c5b51f/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/c8179d49b4feac3af072c6dddf8e1fc10bc109adfd9524a432d09fa84af0f04e/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/94bba44163e922b57ca14ae1423a0b1204bb5732d5255adfd19953330dd5cdf2/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/ee39b36ae6cdbe8af178df470f649476b720864e035b8401adddf72ebf3cf586/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/8df229ff9f93870041e312a1de246875e314b62c3f7a18ad8d750aaddebd7a16/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/d32da5f41cc41a56981872955c4d3b24edaf581a2890c221025cecb09e695b85/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/74fe7b812f0e9a5bcc219545c0b34af1999c756ed3777bd9c733fabcb380d0ef/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/58e51b87ab4479288d40e63af9b15939cfc010eee4f12fc7877babe381f10868/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/58253c2d557d91b0e349b84ec26cc15bcdcb45894ef6db3cff2062b72b68470f/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/dcd27630c4ba35484cdb681bf7df436f130dabfc4829abdf04bf5c4a03db8263/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/c9ed3d19b5f2c31cc5978cf852192be13cb8fe8ec0a367a8a738da20f050c8f1/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/29beffa9ea834aac7430284a56a7c52dcc8c5ac11bc46d0d0fa4c3544825522a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/a10865cd5d8a47d6e7f848611d66fef060842cbc60c9b75d3786dee46198a38d/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/2fe668fdcc0fb2abfbb63af4dcea07a5a37f0708513585b86937f3f6e3d8c4a1/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/eac6409ea55c83facb5d07c5e11b3b6b98ba95881b1abf18b48206635f673323/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/9a3dcb10c90b19ef73d3fa48348ac37a67eac1800e41caf889bf2610003fe477/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/6b1be17eb17ddda68106c33055f5f33125d33952089ac3b74338a41b37899df8/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/5e1bccc7cf1eb2799f6cbf91870d10c9069c39ef960a711611a6db45d1f052ed/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/6a9fed027b400532d36e72c33272b110951f355350996ac493deb510813d2aa9/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 22:08:22.094002 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.093643    4947 manager.go:217] Machine: {Timestamp:2026-10-30 22:08:22.09274827 +0000 UTC m=+0.092095828 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3066216 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f0d7b59cb9095c25be4cfad65344c SystemUUID:ec2f0d7b-59cb-9095-c25b-e4cfad65344c BootID:63d5c887-70ed-4365-9d07-f49adfb96172 Filesystems:[{Device:/var/lib/kubelet/pods/b4d3f83c-7808-4a9a-84d1-d9428d5bd592/volumes/kubernetes.io~projected/kube-api-access-p4ww6 DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35f92cea187b4b04177b4d09b48957700237301a7d7abafbb402a40cd1305055/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a30486-a1f5-4404-8303-6efed2dad370/volumes/kubernetes.io~projected/kube-api-access-dsdc4 DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb5baa5b-c887-42ea-b759-9347ae332051/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3dd55d0f0e7c3edaecbfd56c26bd17cf2d17274eae69e3aa8d6092c185224e4d/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/79823d82-b501-47c9-acae-667e192d721b/volumes/kubernetes.io~projected/kube-api-access-gkbv8 DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8969718f-1dae-4bde-81df-592ee58a1f76/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93cebfd8e1701b6f66ce8b91ee9f0e4ad453336d4fdee5ac05f97d618e9607f1/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcb4abba-e59f-4e64-af78-f1e60697b4b2/volumes/kubernetes.io~projected/kube-api-access-mtpfd DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/610534ce-c054-41fd-8ab6-b70696b7e360/volumes/kubernetes.io~projected/kube-api-access-v7n8j DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6dd646d0a376f2b2fcfcdb457563de9213574005035152738337b593023b105c/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93f981a6-4af6-4405-aec6-89e539c42c67/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7048b86e-5ca2-46c8-8b2d-79116829faf2/volumes/kubernetes.io~projected/kube-api-access-p4s4h DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7d098f8f-7073-4413-9add-d07ef7ce41e1/volumes/kubernetes.io~projected/kube-api-access-trn7v DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/955876a99ad96b940d6504588c872a5ee075b62107954327eb2b6b1f5a086035/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d137caaea3f768787aeb03e28ac7f21294d6e608030e0c363d6939b4c46519f/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9d3d620-4040-45ca-86da-ce6ba161ba3a/volumes/kubernetes.io~projected/kube-api-access-j7s7n DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc134333d3bbaed5e72f2b58f271684b858ac448fa30ce00164b295bfae297f3/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2188065c-aa72-45ea-a0bb-9665fe8f51a2/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-153 DeviceMajor:0 DeviceMinor:153 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76b48563-d38b-4e30-af57-c37cc7b66a9e/volumes/kubernetes.io~projected/kube-api-access-n89g8 DeviceMajor:0 DeviceMinor:329 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94490976-1f2d-4db5-980a-322cad7c057b/volumes/kubernetes.io~projected/kube-api-access-vppkm DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d/volumes/kubernetes.io~projected/kube-api-access-fx7tp DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/da3555d3-b636-4047-957a-632dd893372e/volumes/kubernetes.io~projected/kube-api-access-64d4l DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ed222783bdd148dbf9df772ddd6dae9cb919c817293fa2dc74f531b1e2c093a/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4bf351d84cf28fc3d5c700f4d9c5dbe3275988880a55fa29234c6e3e1b99d073/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/94490976-1f2d-4db5-980a-322cad7c057b/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34feb426af22aebf71fd4d331f0f8b048ce191d95ae6ee3c1ae6cc991c4a706a/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4254134-79fd-411a-a492-3e620d223ddf/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8969718f-1dae-4bde-81df-592ee58a1f76/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/09b48430-dd49-439d-b317-384408fec509/volumes/kubernetes.io~projected/kube-api-access-f8kpl DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd/volumes/kubernetes.io~projected/kube-api-access-j2q5s DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/00c23fb6-cbfb-4498-ad17-15998e6d2d66/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c1575a8-9288-4779-aab6-e5409b097e9f/volumes/kubernetes.io~projected/kube-api-access-nskz5 DeviceMajor:0 DeviceMinor:287 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a30486-a1f5-4404-8303-6efed2dad370/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bb5baa5b-c887-42ea-b759-9347ae332051/volumes/kubernetes.io~projected/kube-api-access-9vss5 DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fdae7310-568d-48ec-80e7-841bd98ab04e/volumes/kubernetes.io~projected/kube-api-access-jz4c8 DeviceMajor:0 DeviceMinor:316 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33a5dce8aae11c140624982e5f02ee86456363d0feae26297084558448dd7501/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e059cdea8af69c2f35c6f2c86433cdfc163872947514209381afd25293ce48d/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/557fc88c-2102-45ec-adc4-80bae4f140cb/volumes/kubernetes.io~projected/kube-api-access-h4pmg DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/00c23fb6-cbfb-4498-ad17-15998e6d2d66/volumes/kubernetes.io~projected/kube-api-access-b8nn2 DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b61d4362c209461fe92640752244e15801e90a93e0168e7597b947a56a628c7/userdata/shm DeviceMajor:0 DeviceMinor:52 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33577b627b4a1735308d81e8e822ca8e9bbf7a1d1c2f3477dc5c6d6aa34ca360/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d81c8448-4580-4174-a7d8-28f5aadf238f/volumes/kubernetes.io~projected/kube-api-access-vkwmz DeviceMajor:0 DeviceMinor:240 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d81c8448-4580-4174-a7d8-28f5aadf238f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:234 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8969718f-1dae-4bde-81df-592ee58a1f76/volumes/kubernetes.io~projected/kube-api-access-rwmpt DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a30486-a1f5-4404-8303-6efed2dad370/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79823d82-b501-47c9-acae-667e192d721b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/ba0fc359-3a19-49ea-b798-2ca0080b30b9/volumes/kubernetes.io~projected/kube-api-access-8mkwp DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a4254134-79fd-411a-a492-3e620d223ddf/volumes/kubernetes.io~projected/kube-api-access-l5cv2 DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f92ebb16-2530-4f59-98d6-c3c2695ad9ba/volumes/kubernetes.io~projected/kube-api-access-xllgn DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f1cdb7881fbb080453f7bb6c53fbac80537ed17128b473b464984c2a6e4db5c/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f41d8d95231e5c3bf2162b438bdae00456b0e9b7d93677891f6adff38147e21c/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f94e750fc006efb749803797e006255ddf0491f5f89e99505b03a099ef8f8b7/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8969718f-1dae-4bde-81df-592ee58a1f76/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3788135a-d420-4738-a292-845c50e486a7/volumes/kubernetes.io~projected/kube-api-access-6c4bd DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/619bd399d0079020184e9e977bac7ce21130ffd14b03696dd52ce4a212f73aa0/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/931bbe7eba22c56243937d2b9926956457d6ca1dc097a15c502a24c087717a23/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bec12d9c-76a5-463a-991e-ca82c18fccd2/volumes/kubernetes.io~projected/kube-api-access-jgn2d DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94490976-1f2d-4db5-980a-322cad7c057b/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91f886b0105538a43484d74538321d69e1c80487ecb28ab205584d477e690228/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0eada557aeb34035fd734d4b685481c39bebcff32b33ee74351b9513eb3e22a/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/79823d82-b501-47c9-acae-667e192d721b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/41dcf151-1d72-4978-b0aa-2046f213cb7e/volumes/kubernetes.io~projected/kube-api-access-rgr7f DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0fe66d1f732276bd45a6520e5b59a121b279c81db215d74ca0ef78832325c8b/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1502caa923a2b2ab695fb6b2b2f7ef5d1b4e3b7dc87fba00ba473f880ead9d4a/userdata/shm DeviceMajor:0 DeviceMinor:252 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4369922-ceac-4aaf-9a8b-35c5a0f27d91/volumes/kubernetes.io~projected/kube-api-access-jl8s9 DeviceMajor:0 DeviceMinor:231 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09b48430-dd49-439d-b317-384408fec509/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1502caa923a2b2a MacAddress:aa:3e:b7:19:1e:fb Speed:10000 Mtu:8901} {Name:1e059cdea8af69c MacAddress:f2:98:27:c5:dc:f6 Speed:10000 Mtu:8901} {Name:1f94e750fc006ef MacAddress:32:00:5e:89:4b:06 Speed:10000 Mtu:8901} {Name:34feb426af22aeb MacAddress:16:dc:9a:82:5f:ab Speed:10000 Mtu:8901} {Name:35f92cea187b4b0 MacAddress:92:cb:8a:8b:5a:2d Speed:10000 Mtu:8901} {Name:3dd55d0f0e7c3ed MacAddress:12:70:f6:b1:df:28 Speed:10000 Mtu:8901} {Name:4bf351d84cf28fc MacAddress:12:cb:40:1b:4f:b1 Speed:10000 Mtu:8901} {Name:6dd646d0a376f2b MacAddress:92:cd:22:4d:62:01 Speed:10000 Mtu:8901} {Name:931bbe7eba22c56 MacAddress:ee:74:48:33:e2:d7 Speed:10000 Mtu:8901} {Name:955876a99ad96b9 MacAddress:b2:56:1c:73:f0:76 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:0a:42:03:3c:7d Speed:0 Mtu:9001} {Name:br-int MacAddress:8a:cf:88:83:76:94 Speed:0 Mtu:8901} {Name:c0eada557aeb340 MacAddress:a6:bb:7e:32:e6:20 Speed:10000 Mtu:8901} {Name:dc134333d3bbaed MacAddress:62:f6:dd:a7:32:44 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:0a:42:03:3c:7d Speed:0 Mtu:9001} {Name:f41d8d95231e5c3 MacAddress:4e:c9:4c:b1:10:90 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:da:c1:0f:e0:61:8f Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:9a:92:69:c2:2c:f3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 22:08:22.094172 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.094008    4947 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 22:08:22.094172 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.094056    4947 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 22:08:22.094363 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.094317    4947 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 22:08:22.094485 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.094364    4947 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-218.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 22:08:22.094526 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.094488    4947 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 22:08:22.094526 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.094495    4947 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 22:08:22.094526 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.094505    4947 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:08:22.094526 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.094523    4947 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:08:22.094630 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.094542    4947 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 22:08:22.094702 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.094693    4947 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 22:08:22.094805 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.094797    4947 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 22:08:22.094840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.094833    4947 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 22:08:22.094874 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.094847    4947 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 22:08:22.094874 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.094857    4947 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 22:08:22.094874 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.094865    4947 kubelet.go:405] "Adding apiserver pod source"
Oct 30 22:08:22.094874 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.094873    4947 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 22:08:22.095713 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.095700    4947 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 22:08:22.095759 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.095714    4947 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:08:22.096173 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.096156    4947 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 22:08:22.109829 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.109812    4947 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 22:08:22.109829 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.109827    4947 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:08:22.109941 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.109918    4947 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 22:08:22.110106 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.110095    4947 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 22:08:22.110135 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.110124    4947 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 22:08:22.110218 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.110209    4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 22:08:22.110250 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.110224    4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 22:08:22.110250 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.110231    4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 22:08:22.110250 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.110237    4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 22:08:22.110250 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.110242    4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 22:08:22.110250 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.110249    4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 22:08:22.110400 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.110256    4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 22:08:22.110400 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.110262    4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 22:08:22.110400 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.110268    4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 22:08:22.110400 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.110274    4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 22:08:22.110400 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.110288    4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 22:08:22.110400 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.110297    4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 22:08:22.110400 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.110326    4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 22:08:22.110400 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.110348    4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 22:08:22.113793 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.113780    4947 server.go:1263] "Started kubelet"
Oct 30 22:08:22.116087 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.113866    4947 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 22:08:22.116087 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.113898    4947 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 22:08:22.116087 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.114019    4947 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 22:08:22.116087 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.114562    4947 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 22:08:22.116087 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.115923    4947 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 22:08:22.114410 ip-10-0-133-218 systemd[1]: Started Kubernetes Kubelet.
Oct 30 22:08:22.121273 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.121169    4947 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 22:08:22.121273 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.121198    4947 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 22:08:22.121273 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.121245    4947 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 22:02:41 +0000 UTC" deadline="2028-04-29 04:30:44.016819668 +0000 UTC"
Oct 30 22:08:22.121273 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.121262    4947 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13110h22m21.895560241s"
Oct 30 22:08:22.122097 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.121298    4947 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 22:08:22.122097 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:22.121292    4947 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-218.us-west-1.compute.internal\" not found"
Oct 30 22:08:22.122097 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.121316    4947 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 22:08:22.122097 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.121429    4947 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 22:08:22.123803 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123775    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" volumeName="kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 22:08:22.123803 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123803    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7048b86e-5ca2-46c8-8b2d-79116829faf2" volumeName="kubernetes.io/configmap/7048b86e-5ca2-46c8-8b2d-79116829faf2-iptables-alerter-script" seLinuxMountContext=""
Oct 30 22:08:22.123925 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123813    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8969718f-1dae-4bde-81df-592ee58a1f76" volumeName="kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-kube-api-access-rwmpt" seLinuxMountContext=""
Oct 30 22:08:22.123925 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123821    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bec12d9c-76a5-463a-991e-ca82c18fccd2" volumeName="kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 22:08:22.123925 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123829    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" volumeName="kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-kube-api-access-fx7tp" seLinuxMountContext=""
Oct 30 22:08:22.123925 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123840    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8969718f-1dae-4bde-81df-592ee58a1f76" volumeName="kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 22:08:22.123925 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123850    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4254134-79fd-411a-a492-3e620d223ddf" volumeName="kubernetes.io/configmap/a4254134-79fd-411a-a492-3e620d223ddf-service-ca-bundle" seLinuxMountContext=""
Oct 30 22:08:22.123925 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123857    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4254134-79fd-411a-a492-3e620d223ddf" volumeName="kubernetes.io/empty-dir/a4254134-79fd-411a-a492-3e620d223ddf-snapshots" seLinuxMountContext=""
Oct 30 22:08:22.123925 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123867    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bec12d9c-76a5-463a-991e-ca82c18fccd2" volumeName="kubernetes.io/projected/bec12d9c-76a5-463a-991e-ca82c18fccd2-kube-api-access-jgn2d" seLinuxMountContext=""
Oct 30 22:08:22.123925 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123878    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" volumeName="kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-bound-sa-token" seLinuxMountContext=""
Oct 30 22:08:22.123925 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123886    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="610534ce-c054-41fd-8ab6-b70696b7e360" volumeName="kubernetes.io/configmap/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-config" seLinuxMountContext=""
Oct 30 22:08:22.123925 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123893    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94490976-1f2d-4db5-980a-322cad7c057b" volumeName="kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-stats-auth" seLinuxMountContext=""
Oct 30 22:08:22.123925 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123900    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4254134-79fd-411a-a492-3e620d223ddf" volumeName="kubernetes.io/configmap/a4254134-79fd-411a-a492-3e620d223ddf-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 22:08:22.123925 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123921    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bec12d9c-76a5-463a-991e-ca82c18fccd2" volumeName="kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-cni-binary-copy" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123936    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" volumeName="kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-certificates" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123944    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7048b86e-5ca2-46c8-8b2d-79116829faf2" volumeName="kubernetes.io/projected/7048b86e-5ca2-46c8-8b2d-79116829faf2-kube-api-access-p4s4h" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123951    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb5baa5b-c887-42ea-b759-9347ae332051" volumeName="kubernetes.io/secret/bb5baa5b-c887-42ea-b759-9347ae332051-serving-cert" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123960    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00c23fb6-cbfb-4498-ad17-15998e6d2d66" volumeName="kubernetes.io/configmap/00c23fb6-cbfb-4498-ad17-15998e6d2d66-config" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123967    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41dcf151-1d72-4978-b0aa-2046f213cb7e" volumeName="kubernetes.io/configmap/41dcf151-1d72-4978-b0aa-2046f213cb7e-serviceca" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123975    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00c23fb6-cbfb-4498-ad17-15998e6d2d66" volumeName="kubernetes.io/configmap/00c23fb6-cbfb-4498-ad17-15998e6d2d66-trusted-ca" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123986    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2188065c-aa72-45ea-a0bb-9665fe8f51a2" volumeName="kubernetes.io/secret/2188065c-aa72-45ea-a0bb-9665fe8f51a2-agent-certs" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.123997    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8969718f-1dae-4bde-81df-592ee58a1f76" volumeName="kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-bound-sa-token" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124009    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4254134-79fd-411a-a492-3e620d223ddf" volumeName="kubernetes.io/empty-dir/a4254134-79fd-411a-a492-3e620d223ddf-tmp" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124021    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41dcf151-1d72-4978-b0aa-2046f213cb7e" volumeName="kubernetes.io/projected/41dcf151-1d72-4978-b0aa-2046f213cb7e-kube-api-access-rgr7f" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124032    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8969718f-1dae-4bde-81df-592ee58a1f76" volumeName="kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-registry-certificates" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124043    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00c23fb6-cbfb-4498-ad17-15998e6d2d66" volumeName="kubernetes.io/projected/00c23fb6-cbfb-4498-ad17-15998e6d2d66-kube-api-access-b8nn2" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124054    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8969718f-1dae-4bde-81df-592ee58a1f76" volumeName="kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-trusted-ca" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124065    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8969718f-1dae-4bde-81df-592ee58a1f76" volumeName="kubernetes.io/empty-dir/8969718f-1dae-4bde-81df-592ee58a1f76-ca-trust-extracted" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124082    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94490976-1f2d-4db5-980a-322cad7c057b" volumeName="kubernetes.io/projected/94490976-1f2d-4db5-980a-322cad7c057b-kube-api-access-vppkm" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124094    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94490976-1f2d-4db5-980a-322cad7c057b" volumeName="kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-default-certificate" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124106    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4369922-ceac-4aaf-9a8b-35c5a0f27d91" volumeName="kubernetes.io/configmap/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-config-volume" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124117    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a30486-a1f5-4404-8303-6efed2dad370" volumeName="kubernetes.io/projected/c0a30486-a1f5-4404-8303-6efed2dad370-kube-api-access-dsdc4" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124129    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09b48430-dd49-439d-b317-384408fec509" volumeName="kubernetes.io/projected/09b48430-dd49-439d-b317-384408fec509-kube-api-access-f8kpl" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124141    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" volumeName="kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-trusted-ca" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124153    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79823d82-b501-47c9-acae-667e192d721b" volumeName="kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-ovnkube-config" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124165    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4369922-ceac-4aaf-9a8b-35c5a0f27d91" volumeName="kubernetes.io/projected/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-kube-api-access-jl8s9" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124169    4947 factory.go:153] Registering CRI-O factory
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124177    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4d3f83c-7808-4a9a-84d1-d9428d5bd592" volumeName="kubernetes.io/projected/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-kube-api-access-p4ww6" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124187    4947 factory.go:223] Registration of the crio container factory successfully
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124189    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bec12d9c-76a5-463a-991e-ca82c18fccd2" volumeName="kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124200    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d098f8f-7073-4413-9add-d07ef7ce41e1" volumeName="kubernetes.io/projected/7d098f8f-7073-4413-9add-d07ef7ce41e1-kube-api-access-trn7v" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124213    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8969718f-1dae-4bde-81df-592ee58a1f76" volumeName="kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-installation-pull-secrets" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124225    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4254134-79fd-411a-a492-3e620d223ddf" volumeName="kubernetes.io/projected/a4254134-79fd-411a-a492-3e620d223ddf-kube-api-access-l5cv2" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124236    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb5baa5b-c887-42ea-b759-9347ae332051" volumeName="kubernetes.io/configmap/bb5baa5b-c887-42ea-b759-9347ae332051-config" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124240    4947 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124247    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb5baa5b-c887-42ea-b759-9347ae332051" volumeName="kubernetes.io/projected/bb5baa5b-c887-42ea-b759-9347ae332051-kube-api-access-9vss5" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124259    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdae7310-568d-48ec-80e7-841bd98ab04e" volumeName="kubernetes.io/projected/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-api-access-jz4c8" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124270    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3788135a-d420-4738-a292-845c50e486a7" volumeName="kubernetes.io/projected/3788135a-d420-4738-a292-845c50e486a7-kube-api-access-6c4bd" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124288    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" volumeName="kubernetes.io/empty-dir/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-ca-trust-extracted" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124301    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="610534ce-c054-41fd-8ab6-b70696b7e360" volumeName="kubernetes.io/projected/610534ce-c054-41fd-8ab6-b70696b7e360-kube-api-access-v7n8j" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124312    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79823d82-b501-47c9-acae-667e192d721b" volumeName="kubernetes.io/projected/79823d82-b501-47c9-acae-667e192d721b-kube-api-access-gkbv8" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124250    4947 factory.go:55] Registering systemd factory
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124324    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb4abba-e59f-4e64-af78-f1e60697b4b2" volumeName="kubernetes.io/projected/fcb4abba-e59f-4e64-af78-f1e60697b4b2-kube-api-access-mtpfd" seLinuxMountContext=""
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124358    4947 factory.go:223] Registration of the systemd container factory successfully
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124384    4947 factory.go:103] Registering Raw factory
Oct 30 22:08:22.124399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124402    4947 manager.go:1199] Started watching for new ooms in manager
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124359    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdae7310-568d-48ec-80e7-841bd98ab04e" volumeName="kubernetes.io/empty-dir/fdae7310-568d-48ec-80e7-841bd98ab04e-data-volume" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124551    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2188065c-aa72-45ea-a0bb-9665fe8f51a2" volumeName="kubernetes.io/configmap/2188065c-aa72-45ea-a0bb-9665fe8f51a2-konnectivity-ca" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124596    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="557fc88c-2102-45ec-adc4-80bae4f140cb" volumeName="kubernetes.io/configmap/557fc88c-2102-45ec-adc4-80bae4f140cb-telemetry-config" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124638    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79823d82-b501-47c9-acae-667e192d721b" volumeName="kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124695    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79823d82-b501-47c9-acae-667e192d721b" volumeName="kubernetes.io/secret/79823d82-b501-47c9-acae-667e192d721b-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124711    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f92ebb16-2530-4f59-98d6-c3c2695ad9ba" volumeName="kubernetes.io/empty-dir/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-tmp-dir" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124725    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00c23fb6-cbfb-4498-ad17-15998e6d2d66" volumeName="kubernetes.io/secret/00c23fb6-cbfb-4498-ad17-15998e6d2d66-serving-cert" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124750    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09b48430-dd49-439d-b317-384408fec509" volumeName="kubernetes.io/secret/09b48430-dd49-439d-b317-384408fec509-serving-cert" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124764    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4254134-79fd-411a-a492-3e620d223ddf" volumeName="kubernetes.io/secret/a4254134-79fd-411a-a492-3e620d223ddf-serving-cert" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124776    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4369922-ceac-4aaf-9a8b-35c5a0f27d91" volumeName="kubernetes.io/empty-dir/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-tmp-dir" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124789    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4d3f83c-7808-4a9a-84d1-d9428d5bd592" volumeName="kubernetes.io/configmap/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-daemon-config" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124806    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a30486-a1f5-4404-8303-6efed2dad370" volumeName="kubernetes.io/empty-dir/c0a30486-a1f5-4404-8303-6efed2dad370-etc-tuned" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124817    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d81c8448-4580-4174-a7d8-28f5aadf238f" volumeName="kubernetes.io/secret/d81c8448-4580-4174-a7d8-28f5aadf238f-cert" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124831    4947 manager.go:319] Starting recovery of all containers
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.124829    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3555d3-b636-4047-957a-632dd893372e" volumeName="kubernetes.io/projected/da3555d3-b636-4047-957a-632dd893372e-kube-api-access-64d4l" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.125389    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="557fc88c-2102-45ec-adc4-80bae4f140cb" volumeName="kubernetes.io/projected/557fc88c-2102-45ec-adc4-80bae4f140cb-kube-api-access-h4pmg" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.125432    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76b48563-d38b-4e30-af57-c37cc7b66a9e" volumeName="kubernetes.io/projected/76b48563-d38b-4e30-af57-c37cc7b66a9e-kube-api-access-n89g8" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.125444    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac726c56-0d31-4b0d-ae06-e547b7ba1dbd" volumeName="kubernetes.io/projected/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-kube-api-access-j2q5s" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.125477    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4d3f83c-7808-4a9a-84d1-d9428d5bd592" volumeName="kubernetes.io/configmap/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-cni-binary-copy" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.125494    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9d3d620-4040-45ca-86da-ce6ba161ba3a" volumeName="kubernetes.io/projected/b9d3d620-4040-45ca-86da-ce6ba161ba3a-kube-api-access-j7s7n" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.125511    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f92ebb16-2530-4f59-98d6-c3c2695ad9ba" volumeName="kubernetes.io/projected/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-kube-api-access-xllgn" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.125546    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93f981a6-4af6-4405-aec6-89e539c42c67" volumeName="kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.125603    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba0fc359-3a19-49ea-b798-2ca0080b30b9" volumeName="kubernetes.io/projected/ba0fc359-3a19-49ea-b798-2ca0080b30b9-kube-api-access-8mkwp" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.125624    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d81c8448-4580-4174-a7d8-28f5aadf238f" volumeName="kubernetes.io/projected/d81c8448-4580-4174-a7d8-28f5aadf238f-kube-api-access-vkwmz" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.125665    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09b48430-dd49-439d-b317-384408fec509" volumeName="kubernetes.io/configmap/09b48430-dd49-439d-b317-384408fec509-config" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.125678    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79823d82-b501-47c9-acae-667e192d721b" volumeName="kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-env-overrides" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.125690    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c1575a8-9288-4779-aab6-e5409b097e9f" volumeName="kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.125712    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba52011d-6892-4b4d-ac14-07e8be859d9e" volumeName="kubernetes.io/configmap/ba52011d-6892-4b4d-ac14-07e8be859d9e-nginx-conf" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.125723    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a30486-a1f5-4404-8303-6efed2dad370" volumeName="kubernetes.io/empty-dir/c0a30486-a1f5-4404-8303-6efed2dad370-tmp" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.125735    4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" volumeName="kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-installation-pull-secrets" seLinuxMountContext=""
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.125746    4947 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 22:08:22.126309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.125752    4947 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 22:08:22.128278 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.128243    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/1.log"
Oct 30 22:08:22.131352 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.130498    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/1.log"
Oct 30 22:08:22.150825 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.150790    4947 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 22:08:22.152147 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.152128    4947 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 22:08:22.152261 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.152154    4947 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 22:08:22.152261 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.152178    4947 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 22:08:22.152261 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:22.152218    4947 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 22:08:22.160728 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.160705    4947 generic.go:356] "Generic (PLEG): container finished" podID="bec12d9c-76a5-463a-991e-ca82c18fccd2" containerID="1053e01dd359bb54f56c731349b7ad530e787e8a01f7804f6ab14b2033184fe4" exitCode=0
Oct 30 22:08:22.160728 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.160728    4947 generic.go:356] "Generic (PLEG): container finished" podID="bec12d9c-76a5-463a-991e-ca82c18fccd2" containerID="5465ab5027d6e81a5c3dc426f89ba00960dd4751659748a75449ab0fad9d13dd" exitCode=0
Oct 30 22:08:22.160852 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.160734    4947 generic.go:356] "Generic (PLEG): container finished" podID="bec12d9c-76a5-463a-991e-ca82c18fccd2" containerID="cea493bd953e4555077717d832dbb2d551f2e65bedd9128d0e435d3a57bf7a96" exitCode=0
Oct 30 22:08:22.160852 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.160740    4947 generic.go:356] "Generic (PLEG): container finished" podID="bec12d9c-76a5-463a-991e-ca82c18fccd2" containerID="fc6548419a124109cb9db87dbda7785c87080549d74af09881993c2f2138acbf" exitCode=0
Oct 30 22:08:22.160852 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.160745    4947 generic.go:356] "Generic (PLEG): container finished" podID="bec12d9c-76a5-463a-991e-ca82c18fccd2" containerID="0977680a73d92dfcf9f648d256685332f7a90f6aa64e89c9a345a22288025b8f" exitCode=0
Oct 30 22:08:22.160852 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.160751    4947 generic.go:356] "Generic (PLEG): container finished" podID="bec12d9c-76a5-463a-991e-ca82c18fccd2" containerID="916d1fb5f87795da6823e1ad7108e2933046929528d2f0d9a3d5ae04ef21c116" exitCode=0
Oct 30 22:08:22.182480 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.182429    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/1.log"
Oct 30 22:08:22.182480 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.182463    4947 generic.go:356] "Generic (PLEG): container finished" podID="00c23fb6-cbfb-4498-ad17-15998e6d2d66" containerID="bab56f6146fa081692f161b3f11e5251a5f21f023755d1e5896cb70b10996697" exitCode=255
Oct 30 22:08:22.199621 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.199595    4947 generic.go:356] "Generic (PLEG): container finished" podID="782d33a65ccf759ca199fdbbd0e7fdf9" containerID="b975cf9bfbace210008b5891dbd017aecf592b9d5e73cce7371970976209cbb0" exitCode=0
Oct 30 22:08:22.209775 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.209757    4947 manager.go:324] Recovery completed
Oct 30 22:08:22.222494 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:22.222468    4947 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-218.us-west-1.compute.internal\" not found"
Oct 30 22:08:22.228214 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.228179    4947 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:08:22.229292 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.229277    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:08:22.229373 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.229303    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:08:22.229373 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.229313    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:08:22.232114 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.232084    4947 cpu_manager.go:226] "Starting" policy="none"
Oct 30 22:08:22.232175 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.232128    4947 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 22:08:22.232175 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.232147    4947 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 22:08:22.232231 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.232101    4947 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:08:22.232324 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.232296    4947 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 22:08:22.232431 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.232316    4947 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 22:08:22.232431 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.232358    4947 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 22:08:22.232431 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.232380    4947 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 22:08:22.232431 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.232388    4947 policy_none.go:50] "Start"
Oct 30 22:08:22.232431 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.232397    4947 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 22:08:22.232431 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.232408    4947 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 22:08:22.232606 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.232535    4947 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 22:08:22.232606 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.232543    4947 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 22:08:22.232606 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.232548    4947 policy_none.go:44] "Start"
Oct 30 22:08:22.233040 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.233021    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:08:22.233090 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.233059    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:08:22.233090 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.233075    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:08:22.236836 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.236812    4947 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 22:08:22.240226 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.240214    4947 manager.go:341] "Starting Device Plugin manager"
Oct 30 22:08:22.240318 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:22.240236    4947 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 22:08:22.240318 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.240244    4947 server.go:87] "Starting device plugin registration server"
Oct 30 22:08:22.240507 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.240493    4947 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 22:08:22.240561 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.240507    4947 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 22:08:22.240608 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.240581    4947 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 22:08:22.240710 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.240671    4947 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 22:08:22.240710 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.240706    4947 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 22:08:22.240710 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.240712    4947 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 22:08:22.240912 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.240781    4947 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T22:08:22.240696892Z","UUID":"91a135df-b47e-47f7-9421-adaa95fb18d5","Handler":null,"Name":"","Endpoint":""}
Oct 30 22:08:22.242171 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.242153    4947 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 22:08:22.242254 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.242177    4947 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 22:08:22.242254 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:22.242180    4947 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 22:08:22.242254 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:22.242218    4947 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-218.us-west-1.compute.internal\" not found"
Oct 30 22:08:22.253115 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.253081    4947 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal"]
Oct 30 22:08:22.253204 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.253194    4947 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:08:22.253894 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.253880    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:08:22.253960 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.253904    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:08:22.253960 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.253914    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:08:22.254170 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.254161    4947 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:08:22.254323 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.254310    4947 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:08:22.260421 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.260404    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:08:22.260502 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.260433    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:08:22.260502 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.260448    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:08:22.260502 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.260409    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:08:22.260617 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.260514    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:08:22.260617 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.260533    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:08:22.260719 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:22.260704    4947 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-218.us-west-1.compute.internal\" not found" node="ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.260764 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.260730    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal" event={"ID":"aaa975e9f9cee63edf4d69aef6d66c72","Type":"ContainerStarted","Data":"241ce13091b06c2bc61beaa452ef0892ce36b94f7d72e42b69fcc30c6f650d31"}
Oct 30 22:08:22.260764 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.260758    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal" event={"ID":"aaa975e9f9cee63edf4d69aef6d66c72","Type":"ContainerStarted","Data":"7ed222783bdd148dbf9df772ddd6dae9cb919c817293fa2dc74f531b1e2c093a"}
Oct 30 22:08:22.260857 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.260772    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal" event={"ID":"782d33a65ccf759ca199fdbbd0e7fdf9","Type":"ContainerStarted","Data":"de310eb449e62d2ff9b2677c39ae898ce7c7f25bce046f905921cf0928379996"}
Oct 30 22:08:22.260857 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.260784    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal" event={"ID":"782d33a65ccf759ca199fdbbd0e7fdf9","Type":"ContainerDied","Data":"b975cf9bfbace210008b5891dbd017aecf592b9d5e73cce7371970976209cbb0"}
Oct 30 22:08:22.260857 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.260795    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal" event={"ID":"782d33a65ccf759ca199fdbbd0e7fdf9","Type":"ContainerStarted","Data":"3b61d4362c209461fe92640752244e15801e90a93e0168e7597b947a56a628c7"}
Oct 30 22:08:22.260857 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.260814    4947 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:08:22.261614 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.261600    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:08:22.261679 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.261648    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:08:22.261679 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.261661    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:08:22.261795 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:22.261783    4947 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-218.us-west-1.compute.internal\" not found" node="ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.311749 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.311723    4947 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 22:08:22.312862 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.312844    4947 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:08:22.312953 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.312920    4947 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:08:22.313005 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.312988    4947 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:08:22.322182 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.322163    4947 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.341447 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.341425    4947 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:08:22.342473 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.342458    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:08:22.342561 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.342482    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:08:22.342561 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.342492    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:08:22.342561 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.342531    4947 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.364304 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.364279    4947 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:08:22.400082 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.400064    4947 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:08:22.400167 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:22.400101    4947 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.400167 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.400113    4947 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.412368 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.412349    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/782d33a65ccf759ca199fdbbd0e7fdf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"782d33a65ccf759ca199fdbbd0e7fdf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.412430 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.412373    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/782d33a65ccf759ca199fdbbd0e7fdf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"782d33a65ccf759ca199fdbbd0e7fdf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.412430 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.412390    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aaa975e9f9cee63edf4d69aef6d66c72-config\") pod \"kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"aaa975e9f9cee63edf4d69aef6d66c72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.465764 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.465715    4947 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.465838 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.465780    4947 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.466359 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.466324    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeNotReady"
Oct 30 22:08:22.466401 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.466379    4947 setters.go:546] "Node became not ready" node="ip-10-0-133-218.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T22:08:22Z","lastTransitionTime":"2026-10-30T22:08:22Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 22:08:22.487122 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.487100    4947 nodeinfomanager.go:471] existing CSINode "ip-10-0-133-218.us-west-1.compute.internal" is owned by different node (oldNodeID="8a9c70c7-6313-402c-a814-dfb01e0dfdc2", newNodeID=""), cleaning up...
Oct 30 22:08:22.494731 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.494714    4947 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:08:22.494791 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:22.494747    4947 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.512915 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.512892    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/782d33a65ccf759ca199fdbbd0e7fdf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"782d33a65ccf759ca199fdbbd0e7fdf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.512968 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.512919    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/782d33a65ccf759ca199fdbbd0e7fdf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"782d33a65ccf759ca199fdbbd0e7fdf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.512968 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.512937    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aaa975e9f9cee63edf4d69aef6d66c72-config\") pod \"kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"aaa975e9f9cee63edf4d69aef6d66c72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.513042 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.512973    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aaa975e9f9cee63edf4d69aef6d66c72-config\") pod \"kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"aaa975e9f9cee63edf4d69aef6d66c72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.513042 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.513006    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/782d33a65ccf759ca199fdbbd0e7fdf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"782d33a65ccf759ca199fdbbd0e7fdf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.513042 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.513006    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/782d33a65ccf759ca199fdbbd0e7fdf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal\" (UID: \"782d33a65ccf759ca199fdbbd0e7fdf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:22.738467 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.738404    4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-218.us-west-1.compute.internal" event="NodeReady"
Oct 30 22:08:22.738588 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:22.738489    4947 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 22:08:23.095901 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.095870    4947 apiserver.go:52] "Watching apiserver"
Oct 30 22:08:23.205556 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.205530    4947 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:23.205729 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.205627    4947 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:23.250677 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.250650    4947 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 22:08:23.251464 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.251440    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-xtgwp","openshift-ingress-canary/ingress-canary-svwl7","openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx","openshift-multus/multus-2ptq8","openshift-network-console/networking-console-plugin-86c478bb88-8sk5t","openshift-ovn-kubernetes/ovnkube-node-z2zrp","openshift-insights/insights-runtime-extractor-dqpbw","kube-system/kas-connection-checker-685895c7c9-jktvv","openshift-dns/node-resolver-s2528","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8k87n","kube-system/kas-connection-checker-685895c7c9-79956","kube-system/konnectivity-agent-fgkn6","openshift-image-registry/image-registry-bdbcb7997-ggr5g","openshift-dns/dns-default-jg2dx","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp","openshift-network-diagnostics/network-check-source-6f78d59f99-f8tgh","openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl","openshift-insights/insights-operator-7c4986d449-652qc","kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal","openshift-console-operator/console-operator-7cd56b8dbc-gs4xr","openshift-kube-storage-version-migrator/migrator-66df5d496b-642k8","openshift-cluster-node-tuning-operator/tuned-np6dm","openshift-network-diagnostics/network-check-target-7h6kb","openshift-multus/network-metrics-daemon-g5pws","openshift-image-registry/node-ca-bc87g","openshift-ingress/router-default-56d5748b65-v9kvx","openshift-image-registry/image-registry-5ff86975d4-6xkd8","kube-system/kas-connection-checker-685895c7c9-6dsmb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-c5whj","openshift-multus/multus-additional-cni-plugins-cs9rl","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"]
Oct 30 22:08:23.252847 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.252826    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:23.253124 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.253098    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:23.257666 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.257641    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:23.258653 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.258634    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:23.259024 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.259005    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:23.263409 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.263390    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:23.264404 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.264388    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:23.264947 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.264932    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:23.266010 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.265992    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:23.267380 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.267364    4947 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:08:23.267459 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.267410    4947 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:23.268607 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.268594    4947 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:08:23.268663 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.268627    4947 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-218.us-west-1.compute.internal"
Oct 30 22:08:23.305674 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.305654    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.305817 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.305706    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.306529 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.306514    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 22:08:23.308377 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.308364    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.310465 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.310450    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 22:08:23.310638 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.310618    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 22:08:23.310738 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.310661    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 22:08:23.310738 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.310664    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jnhs4\""
Oct 30 22:08:23.310829 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.310814    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 22:08:23.310829 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.310814    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-r8xph\""
Oct 30 22:08:23.310899 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.310845    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 22:08:23.310899 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.310851    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 22:08:23.311035 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.311018    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 22:08:23.311147 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.311121    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 22:08:23.311147 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.311133    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 22:08:23.311320 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.311302    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-h8kpj\""
Oct 30 22:08:23.311400 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.311368    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-shbx7\""
Oct 30 22:08:23.311465 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.311410    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-snjn2\""
Oct 30 22:08:23.311465 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.311417    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-d2r6m\""
Oct 30 22:08:23.311619 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.311600    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wwvsp\""
Oct 30 22:08:23.311678 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.311639    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4zbp8\""
Oct 30 22:08:23.313478 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.313465    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-r9j8h\""
Oct 30 22:08:23.323486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.323471    4947 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 22:08:23.352698 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.352644    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 22:08:23.352885 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.352870    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 22:08:23.353301 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.353278    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-cdzq2\""
Oct 30 22:08:23.353423 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.353359    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2wgjn\""
Oct 30 22:08:23.353493 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.353441    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-mmvq7\""
Oct 30 22:08:23.353547 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.353530    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 22:08:23.353593 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.353547    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qzh2l\""
Oct 30 22:08:23.353593 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.353536    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-78xqf\""
Oct 30 22:08:23.353684 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.353604    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lds9w\""
Oct 30 22:08:23.353684 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.353620    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-486qv\""
Oct 30 22:08:23.353684 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.353644    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2djd4\""
Oct 30 22:08:23.353811 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.353693    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 22:08:23.353877 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.353864    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-kpsh7\""
Oct 30 22:08:23.354923 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.354906    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 22:08:23.354992 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.354967    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-f5pfp\""
Oct 30 22:08:23.355081 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.355064    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 22:08:23.355575 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.355554    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c8f2h\""
Oct 30 22:08:23.355750 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.355559    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dl5xt\""
Oct 30 22:08:23.355860 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.355764    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 22:08:23.355923 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.355781    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 22:08:23.355923 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.355815    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nff88\""
Oct 30 22:08:23.356169 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.356154    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xjxjz\""
Oct 30 22:08:23.356246 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.356214    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 22:08:23.356478 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.356433    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 22:08:23.356553 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.356521    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s8t4z\""
Oct 30 22:08:23.356553 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.356521    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tmrfp\""
Oct 30 22:08:23.356691 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.356674    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lmcdt\""
Oct 30 22:08:23.356870 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.356850    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 22:08:23.357224 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.357204    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dlplv\""
Oct 30 22:08:23.357313 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.357301    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 22:08:23.405812 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.405788    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 22:08:23.405812 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.405802    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.405952 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.405788    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.405952 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.405826    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 22:08:23.406063 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.405951    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 22:08:23.406063 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.405963    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.406153 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.406075    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 22:08:23.406153 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.406123    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.406241 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.406218    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.406241 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.406222    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.406321 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.406271    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.406448 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.406428    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 22:08:23.406516 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.406431    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.406516 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.406468    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 22:08:23.406604 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.406515    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.406655 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.406590    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.406691 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.406655    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 22:08:23.406876 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.406855    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 22:08:23.406936 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.406878    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.406936 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.406884    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.407024 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.406970    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.407024 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.406974    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.407159 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407145    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.407206 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407165    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.407398 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407381    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.407444 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407383    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.407501 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407491    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.407550 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407506    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.407550 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407506    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.407550 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407509    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 22:08:23.407634 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407572    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.407634 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407603    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 22:08:23.407708 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407697    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 22:08:23.407742 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407710    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.407789 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407779    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.407819 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407786    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 22:08:23.407819 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407782    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.407819 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407802    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 22:08:23.407819 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407810    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 22:08:23.407819 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407815    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.407995 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407840    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.407995 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407926    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.407995 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407943    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.408111 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.407998    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 22:08:23.408111 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.408034    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 22:08:23.408111 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.408044    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.408245 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.408047    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.408245 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.408148    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.408245 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.408160    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:08:23.408245 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.408199    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 22:08:23.408245 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.408217    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 22:08:23.408245 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.408224    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.408245 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.408233    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 22:08:23.408529 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.408259    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 22:08:23.408529 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.408261    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 22:08:23.409180 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.409162    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 22:08:23.409275 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.409264    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 22:08:23.419147 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419131    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-cni-bin\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.419191 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419159    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba52011d-6892-4b4d-ac14-07e8be859d9e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:23.419191 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419184    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7n8j\" (UniqueName: \"kubernetes.io/projected/610534ce-c054-41fd-8ab6-b70696b7e360-kube-api-access-v7n8j\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:23.419255 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419201    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41dcf151-1d72-4978-b0aa-2046f213cb7e-serviceca\") pod \"node-ca-bc87g\" (UID: \"41dcf151-1d72-4978-b0aa-2046f213cb7e\") " pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:08:23.419255 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419247    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-cni-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.419351 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419323    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-host\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.419404 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419390    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-slash\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.419439 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419422    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-systemd\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.419479 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419442    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdae7310-568d-48ec-80e7-841bd98ab04e-crio-socket\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:23.419519 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419493    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba52011d-6892-4b4d-ac14-07e8be859d9e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:23.419519 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419499    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-image-registry-private-configuration\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:23.419594 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419539    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00c23fb6-cbfb-4498-ad17-15998e6d2d66-config\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:23.419655 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419640    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7s7n\" (UniqueName: \"kubernetes.io/projected/b9d3d620-4040-45ca-86da-ce6ba161ba3a-kube-api-access-j7s7n\") pod \"kas-connection-checker-685895c7c9-6dsmb\" (UID: \"b9d3d620-4040-45ca-86da-ce6ba161ba3a\") " pod="kube-system/kas-connection-checker-685895c7c9-6dsmb"
Oct 30 22:08:23.419703 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419675    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41dcf151-1d72-4978-b0aa-2046f213cb7e-serviceca\") pod \"node-ca-bc87g\" (UID: \"41dcf151-1d72-4978-b0aa-2046f213cb7e\") " pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:08:23.419771 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419739    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-image-registry-private-configuration\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:23.419771 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419753    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4ww6\" (UniqueName: \"kubernetes.io/projected/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-kube-api-access-p4ww6\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.419875 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419821    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-var-lib-kubelet\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.419875 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419852    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93f981a6-4af6-4405-aec6-89e539c42c67-kubelet-config\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:08:23.419875 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419854    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00c23fb6-cbfb-4498-ad17-15998e6d2d66-config\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:23.420009 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419937    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2q5s\" (UniqueName: \"kubernetes.io/projected/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-kube-api-access-j2q5s\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.420056 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.419992    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtpfd\" (UniqueName: \"kubernetes.io/projected/fcb4abba-e59f-4e64-af78-f1e60697b4b2-kube-api-access-mtpfd\") pod \"kas-connection-checker-685895c7c9-jktvv\" (UID: \"fcb4abba-e59f-4e64-af78-f1e60697b4b2\") " pod="kube-system/kas-connection-checker-685895c7c9-jktvv"
Oct 30 22:08:23.420144 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420125    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2188065c-aa72-45ea-a0bb-9665fe8f51a2-konnectivity-ca\") pod \"konnectivity-agent-fgkn6\" (UID: \"2188065c-aa72-45ea-a0bb-9665fe8f51a2\") " pod="kube-system/konnectivity-agent-fgkn6"
Oct 30 22:08:23.420199 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420170    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-multus-certs\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.420247 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420196    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysconfig\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.420247 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420220    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:23.420357 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420248    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:08:23.420357 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420286    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4s4h\" (UniqueName: \"kubernetes.io/projected/7048b86e-5ca2-46c8-8b2d-79116829faf2-kube-api-access-p4s4h\") pod \"iptables-alerter-c5whj\" (UID: \"7048b86e-5ca2-46c8-8b2d-79116829faf2\") " pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:08:23.420357 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420296    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2188065c-aa72-45ea-a0bb-9665fe8f51a2-konnectivity-ca\") pod \"konnectivity-agent-fgkn6\" (UID: \"2188065c-aa72-45ea-a0bb-9665fe8f51a2\") " pod="kube-system/konnectivity-agent-fgkn6"
Oct 30 22:08:23.420446 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420423    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5cv2\" (UniqueName: \"kubernetes.io/projected/a4254134-79fd-411a-a492-3e620d223ddf-kube-api-access-l5cv2\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:23.420477 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420443    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-lib-modules\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.420477 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420459    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-cni-netd\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.420587 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420569    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93f981a6-4af6-4405-aec6-89e539c42c67-original-pull-secret\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:08:23.420617 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420579    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4254134-79fd-411a-a492-3e620d223ddf-service-ca-bundle\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:23.420644 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420615    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsdc4\" (UniqueName: \"kubernetes.io/projected/c0a30486-a1f5-4404-8303-6efed2dad370-kube-api-access-dsdc4\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.420644 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420638    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-systemd-units\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.420699 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420658    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-ovnkube-script-lib\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.420699 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420674    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbv8\" (UniqueName: \"kubernetes.io/projected/79823d82-b501-47c9-acae-667e192d721b-kube-api-access-gkbv8\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.420762 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420697    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:23.420762 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420674    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4254134-79fd-411a-a492-3e620d223ddf-service-ca-bundle\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:23.420762 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420727    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-cni-binary-copy\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.420882 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420820    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-sys\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.420882 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420852    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0a30486-a1f5-4404-8303-6efed2dad370-tmp\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.420965 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420878    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-run-netns\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.420965 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420903    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdae7310-568d-48ec-80e7-841bd98ab04e-data-volume\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:23.420965 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420903    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-ovnkube-script-lib\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.420965 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420930    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nskz5\" (UniqueName: \"kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5\") pod \"network-check-target-7h6kb\" (UID: \"7c1575a8-9288-4779-aab6-e5409b097e9f\") " pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:08:23.421143 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420949    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0a30486-a1f5-4404-8303-6efed2dad370-tmp\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.421143 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420974    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:23.421143 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.420981    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-cni-binary-copy\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.421143 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421022    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-modprobe-d\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.421143 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421036    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdae7310-568d-48ec-80e7-841bd98ab04e-data-volume\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:23.421143 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421051    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-systemd\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.421143 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421075    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-ovnkube-config\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.421143 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421089    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-cnibin\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.421143 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421106    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0a30486-a1f5-4404-8303-6efed2dad370-etc-tuned\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.421143 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421131    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl8s9\" (UniqueName: \"kubernetes.io/projected/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-kube-api-access-jl8s9\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:23.421588 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421162    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vss5\" (UniqueName: \"kubernetes.io/projected/bb5baa5b-c887-42ea-b759-9347ae332051-kube-api-access-9vss5\") pod \"service-ca-operator-544d749c8c-js4z7\" (UID: \"bb5baa5b-c887-42ea-b759-9347ae332051\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"
Oct 30 22:08:23.421588 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421250    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-ovnkube-config\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.421588 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421351    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0a30486-a1f5-4404-8303-6efed2dad370-etc-tuned\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.421588 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421381    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8nn2\" (UniqueName: \"kubernetes.io/projected/00c23fb6-cbfb-4498-ad17-15998e6d2d66-kube-api-access-b8nn2\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:23.421588 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421404    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2188065c-aa72-45ea-a0bb-9665fe8f51a2-agent-certs\") pod \"konnectivity-agent-fgkn6\" (UID: \"2188065c-aa72-45ea-a0bb-9665fe8f51a2\") " pod="kube-system/konnectivity-agent-fgkn6"
Oct 30 22:08:23.421588 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421473    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4254134-79fd-411a-a492-3e620d223ddf-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:23.421588 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421493    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-k8s-cni-cncf-io\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.421588 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421513    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-cni-multus\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.421588 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421538    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-node-log\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.421588 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421573    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-bound-sa-token\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421599    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/557fc88c-2102-45ec-adc4-80bae4f140cb-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421623    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xllgn\" (UniqueName: \"kubernetes.io/projected/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-kube-api-access-xllgn\") pod \"node-resolver-s2528\" (UID: \"f92ebb16-2530-4f59-98d6-c3c2695ad9ba\") " pod="openshift-dns/node-resolver-s2528"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421649    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-socket-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421661    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2188065c-aa72-45ea-a0bb-9665fe8f51a2-agent-certs\") pod \"konnectivity-agent-fgkn6\" (UID: \"2188065c-aa72-45ea-a0bb-9665fe8f51a2\") " pod="kube-system/konnectivity-agent-fgkn6"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421672    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-conf-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421697    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-trusted-ca\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421714    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421730    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421760    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4254134-79fd-411a-a492-3e620d223ddf-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421804    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trn7v\" (UniqueName: \"kubernetes.io/projected/7d098f8f-7073-4413-9add-d07ef7ce41e1-kube-api-access-trn7v\") pod \"network-check-source-6f78d59f99-f8tgh\" (UID: \"7d098f8f-7073-4413-9add-d07ef7ce41e1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f8tgh"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421838    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-cni-bin\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421873    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgn2d\" (UniqueName: \"kubernetes.io/projected/bec12d9c-76a5-463a-991e-ca82c18fccd2-kube-api-access-jgn2d\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421915    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-default-certificate\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421923    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/557fc88c-2102-45ec-adc4-80bae4f140cb-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421947    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421972    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421979    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.421997    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:23.422034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422037    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-tmp-dir\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422071    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-env-overrides\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422140    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-tmp-dir\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422132    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8969718f-1dae-4bde-81df-592ee58a1f76-ca-trust-extracted\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422140    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-default-certificate\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422186    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422229    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8969718f-1dae-4bde-81df-592ee58a1f76-ca-trust-extracted\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422216    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a4254134-79fd-411a-a492-3e620d223ddf-snapshots\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422266    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/79823d82-b501-47c9-acae-667e192d721b-env-overrides\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422276    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgr7f\" (UniqueName: \"kubernetes.io/projected/41dcf151-1d72-4978-b0aa-2046f213cb7e-kube-api-access-rgr7f\") pod \"node-ca-bc87g\" (UID: \"41dcf151-1d72-4978-b0aa-2046f213cb7e\") " pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422293    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a4254134-79fd-411a-a492-3e620d223ddf-snapshots\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422309    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mkwp\" (UniqueName: \"kubernetes.io/projected/ba0fc359-3a19-49ea-b798-2ca0080b30b9-kube-api-access-8mkwp\") pod \"kas-connection-checker-685895c7c9-79956\" (UID: \"ba0fc359-3a19-49ea-b798-2ca0080b30b9\") " pod="kube-system/kas-connection-checker-685895c7c9-79956"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422363    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7048b86e-5ca2-46c8-8b2d-79116829faf2-iptables-alerter-script\") pod \"iptables-alerter-c5whj\" (UID: \"7048b86e-5ca2-46c8-8b2d-79116829faf2\") " pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422392    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7048b86e-5ca2-46c8-8b2d-79116829faf2-host-slash\") pod \"iptables-alerter-c5whj\" (UID: \"7048b86e-5ca2-46c8-8b2d-79116829faf2\") " pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422425    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-etc-selinux\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422458    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-system-cni-dir\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422487    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41dcf151-1d72-4978-b0aa-2046f213cb7e-host\") pod \"node-ca-bc87g\" (UID: \"41dcf151-1d72-4978-b0aa-2046f213cb7e\") " pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422518    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-socket-dir-parent\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422559    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-etc-kubernetes\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422598    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-trusted-ca\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422609    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb5baa5b-c887-42ea-b759-9347ae332051-config\") pod \"service-ca-operator-544d749c8c-js4z7\" (UID: \"bb5baa5b-c887-42ea-b759-9347ae332051\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422632    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysctl-conf\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.422840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422799    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb5baa5b-c887-42ea-b759-9347ae332051-config\") pod \"service-ca-operator-544d749c8c-js4z7\" (UID: \"bb5baa5b-c887-42ea-b759-9347ae332051\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"
Oct 30 22:08:23.423745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422854    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-var-lib-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.423745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422886    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-image-registry-private-configuration\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:23.423745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.422938    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64d4l\" (UniqueName: \"kubernetes.io/projected/da3555d3-b636-4047-957a-632dd893372e-kube-api-access-64d4l\") pod \"volume-data-source-validator-58c9f5f46d-8k87n\" (UID: \"da3555d3-b636-4047-957a-632dd893372e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8k87n"
Oct 30 22:08:23.423745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423050    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-tmp-dir\") pod \"node-resolver-s2528\" (UID: \"f92ebb16-2530-4f59-98d6-c3c2695ad9ba\") " pod="openshift-dns/node-resolver-s2528"
Oct 30 22:08:23.423745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423140    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-image-registry-private-configuration\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:23.423745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423146    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7048b86e-5ca2-46c8-8b2d-79116829faf2-iptables-alerter-script\") pod \"iptables-alerter-c5whj\" (UID: \"7048b86e-5ca2-46c8-8b2d-79116829faf2\") " pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:08:23.423745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423292    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-log-socket\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.423745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423296    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-tmp-dir\") pod \"node-resolver-s2528\" (UID: \"f92ebb16-2530-4f59-98d6-c3c2695ad9ba\") " pod="openshift-dns/node-resolver-s2528"
Oct 30 22:08:23.423745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423363    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-run-ovn-kubernetes\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.423745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423397    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4254134-79fd-411a-a492-3e620d223ddf-serving-cert\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:23.423745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423464    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:23.423745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423533    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4254134-79fd-411a-a492-3e620d223ddf-serving-cert\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:23.423745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423533    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:23.423745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423583    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-stats-auth\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:23.423745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423621    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-etc-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.423745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423626    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:23.423745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423645    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/79823d82-b501-47c9-acae-667e192d721b-ovn-node-metrics-cert\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423778    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/79823d82-b501-47c9-acae-667e192d721b-ovn-node-metrics-cert\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423780    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-stats-auth\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423825    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09b48430-dd49-439d-b317-384408fec509-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp\" (UID: \"09b48430-dd49-439d-b317-384408fec509\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423870    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423913    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-kubelet\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423941    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysctl-d\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.423966    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb5baa5b-c887-42ea-b759-9347ae332051-serving-cert\") pod \"service-ca-operator-544d749c8c-js4z7\" (UID: \"bb5baa5b-c887-42ea-b759-9347ae332051\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424008    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-sys-fs\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424013    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09b48430-dd49-439d-b317-384408fec509-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp\" (UID: \"09b48430-dd49-439d-b317-384408fec509\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424088    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d81c8448-4580-4174-a7d8-28f5aadf238f-cert\") pod \"ingress-canary-svwl7\" (UID: \"d81c8448-4580-4174-a7d8-28f5aadf238f\") " pod="openshift-ingress-canary/ingress-canary-svwl7"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424100    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb5baa5b-c887-42ea-b759-9347ae332051-serving-cert\") pod \"service-ca-operator-544d749c8c-js4z7\" (UID: \"bb5baa5b-c887-42ea-b759-9347ae332051\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424142    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-run\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424175    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8kpl\" (UniqueName: \"kubernetes.io/projected/09b48430-dd49-439d-b317-384408fec509-kube-api-access-f8kpl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp\" (UID: \"09b48430-dd49-439d-b317-384408fec509\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424215    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424265    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d81c8448-4580-4174-a7d8-28f5aadf238f-cert\") pod \"ingress-canary-svwl7\" (UID: \"d81c8448-4580-4174-a7d8-28f5aadf238f\") " pod="openshift-ingress-canary/ingress-canary-svwl7"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424276    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4c8\" (UniqueName: \"kubernetes.io/projected/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-api-access-jz4c8\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424319    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4pmg\" (UniqueName: \"kubernetes.io/projected/557fc88c-2102-45ec-adc4-80bae4f140cb-kube-api-access-h4pmg\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424372    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00c23fb6-cbfb-4498-ad17-15998e6d2d66-trusted-ca\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424398    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-ovn\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424424    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7tp\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-kube-api-access-fx7tp\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424450    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-system-cni-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424479    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-daemon-config\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424504    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-bound-sa-token\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424532    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-device-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424554    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-kubelet\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.424583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424586    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-installation-pull-secrets\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424614    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424623    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00c23fb6-cbfb-4498-ad17-15998e6d2d66-trusted-ca\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424641    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-certificates\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424678    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-installation-pull-secrets\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424721    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-daemon-config\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424726    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-config-volume\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424825    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c4bd\" (UniqueName: \"kubernetes.io/projected/3788135a-d420-4738-a292-845c50e486a7-kube-api-access-6c4bd\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424838    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-installation-pull-secrets\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424851    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-certificates\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424882    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93f981a6-4af6-4405-aec6-89e539c42c67-dbus\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424896    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-installation-pull-secrets\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424914    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-ca-trust-extracted\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424939    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-config-volume\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424960    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-cnibin\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424985    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-ca-trust-extracted\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.424987    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-cni-binary-copy\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425093    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425111    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00c23fb6-cbfb-4498-ad17-15998e6d2d66-serving-cert\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425127    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-netns\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425130    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-cni-binary-copy\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425142    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-hostroot\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425158    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425176    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n89g8\" (UniqueName: \"kubernetes.io/projected/76b48563-d38b-4e30-af57-c37cc7b66a9e-kube-api-access-n89g8\") pod \"migrator-66df5d496b-642k8\" (UID: \"76b48563-d38b-4e30-af57-c37cc7b66a9e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-642k8"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425196    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425242    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bec12d9c-76a5-463a-991e-ca82c18fccd2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425249    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4254134-79fd-411a-a492-3e620d223ddf-tmp\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425242    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00c23fb6-cbfb-4498-ad17-15998e6d2d66-serving-cert\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425275    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-hosts-file\") pod \"node-resolver-s2528\" (UID: \"f92ebb16-2530-4f59-98d6-c3c2695ad9ba\") " pod="openshift-dns/node-resolver-s2528"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425301    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-kubernetes\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425306    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4254134-79fd-411a-a492-3e620d223ddf-tmp\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425361    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09b48430-dd49-439d-b317-384408fec509-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp\" (UID: \"09b48430-dd49-439d-b317-384408fec509\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425383    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-registration-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425400    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkwmz\" (UniqueName: \"kubernetes.io/projected/d81c8448-4580-4174-a7d8-28f5aadf238f-kube-api-access-vkwmz\") pod \"ingress-canary-svwl7\" (UID: \"d81c8448-4580-4174-a7d8-28f5aadf238f\") " pod="openshift-ingress-canary/ingress-canary-svwl7"
Oct 30 22:08:23.425463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425415    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vppkm\" (UniqueName: \"kubernetes.io/projected/94490976-1f2d-4db5-980a-322cad7c057b-kube-api-access-vppkm\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:23.426384 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425513    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-registry-certificates\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:23.426384 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425538    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-trusted-ca\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:23.426384 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425554    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:08:23.426384 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425574    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwmpt\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-kube-api-access-rwmpt\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:23.426384 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425605    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09b48430-dd49-439d-b317-384408fec509-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp\" (UID: \"09b48430-dd49-439d-b317-384408fec509\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"
Oct 30 22:08:23.426384 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425656    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-registry-certificates\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:23.426384 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.425868    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-trusted-ca\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:23.480212 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.480186    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7n8j\" (UniqueName: \"kubernetes.io/projected/610534ce-c054-41fd-8ab6-b70696b7e360-kube-api-access-v7n8j\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:23.480356 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.480186    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7s7n\" (UniqueName: \"kubernetes.io/projected/b9d3d620-4040-45ca-86da-ce6ba161ba3a-kube-api-access-j7s7n\") pod \"kas-connection-checker-685895c7c9-6dsmb\" (UID: \"b9d3d620-4040-45ca-86da-ce6ba161ba3a\") " pod="kube-system/kas-connection-checker-685895c7c9-6dsmb"
Oct 30 22:08:23.480356 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.480251    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5cv2\" (UniqueName: \"kubernetes.io/projected/a4254134-79fd-411a-a492-3e620d223ddf-kube-api-access-l5cv2\") pod \"insights-operator-7c4986d449-652qc\" (UID: \"a4254134-79fd-411a-a492-3e620d223ddf\") " pod="openshift-insights/insights-operator-7c4986d449-652qc"
Oct 30 22:08:23.480356 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.480237    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4s4h\" (UniqueName: \"kubernetes.io/projected/7048b86e-5ca2-46c8-8b2d-79116829faf2-kube-api-access-p4s4h\") pod \"iptables-alerter-c5whj\" (UID: \"7048b86e-5ca2-46c8-8b2d-79116829faf2\") " pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:08:23.480795 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.480760    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4ww6\" (UniqueName: \"kubernetes.io/projected/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-kube-api-access-p4ww6\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.480870 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.480851    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsdc4\" (UniqueName: \"kubernetes.io/projected/c0a30486-a1f5-4404-8303-6efed2dad370-kube-api-access-dsdc4\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.481348 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.481312    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vss5\" (UniqueName: \"kubernetes.io/projected/bb5baa5b-c887-42ea-b759-9347ae332051-kube-api-access-9vss5\") pod \"service-ca-operator-544d749c8c-js4z7\" (UID: \"bb5baa5b-c887-42ea-b759-9347ae332051\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7"
Oct 30 22:08:23.481348 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.481316    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl8s9\" (UniqueName: \"kubernetes.io/projected/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-kube-api-access-jl8s9\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:23.481519 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.481498    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nskz5\" (UniqueName: \"kubernetes.io/projected/7c1575a8-9288-4779-aab6-e5409b097e9f-kube-api-access-nskz5\") pod \"network-check-target-7h6kb\" (UID: \"7c1575a8-9288-4779-aab6-e5409b097e9f\") " pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:08:23.481572 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.481559    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtpfd\" (UniqueName: \"kubernetes.io/projected/fcb4abba-e59f-4e64-af78-f1e60697b4b2-kube-api-access-mtpfd\") pod \"kas-connection-checker-685895c7c9-jktvv\" (UID: \"fcb4abba-e59f-4e64-af78-f1e60697b4b2\") " pod="kube-system/kas-connection-checker-685895c7c9-jktvv"
Oct 30 22:08:23.481872 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.481857    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trn7v\" (UniqueName: \"kubernetes.io/projected/7d098f8f-7073-4413-9add-d07ef7ce41e1-kube-api-access-trn7v\") pod \"network-check-source-6f78d59f99-f8tgh\" (UID: \"7d098f8f-7073-4413-9add-d07ef7ce41e1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f8tgh"
Oct 30 22:08:23.482076 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.482062    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgn2d\" (UniqueName: \"kubernetes.io/projected/bec12d9c-76a5-463a-991e-ca82c18fccd2-kube-api-access-jgn2d\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:08:23.482641 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.482626    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-bound-sa-token\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:23.482754 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.482739    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2q5s\" (UniqueName: \"kubernetes.io/projected/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-kube-api-access-j2q5s\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.485321 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.485301    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xllgn\" (UniqueName: \"kubernetes.io/projected/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-kube-api-access-xllgn\") pod \"node-resolver-s2528\" (UID: \"f92ebb16-2530-4f59-98d6-c3c2695ad9ba\") " pod="openshift-dns/node-resolver-s2528"
Oct 30 22:08:23.485401 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.485387    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8nn2\" (UniqueName: \"kubernetes.io/projected/00c23fb6-cbfb-4498-ad17-15998e6d2d66-kube-api-access-b8nn2\") pod \"console-operator-7cd56b8dbc-gs4xr\" (UID: \"00c23fb6-cbfb-4498-ad17-15998e6d2d66\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:23.485709 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.485694    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkbv8\" (UniqueName: \"kubernetes.io/projected/79823d82-b501-47c9-acae-667e192d721b-kube-api-access-gkbv8\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.486742 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.486719    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64d4l\" (UniqueName: \"kubernetes.io/projected/da3555d3-b636-4047-957a-632dd893372e-kube-api-access-64d4l\") pod \"volume-data-source-validator-58c9f5f46d-8k87n\" (UID: \"da3555d3-b636-4047-957a-632dd893372e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8k87n"
Oct 30 22:08:23.486840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.486820    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n89g8\" (UniqueName: \"kubernetes.io/projected/76b48563-d38b-4e30-af57-c37cc7b66a9e-kube-api-access-n89g8\") pod \"migrator-66df5d496b-642k8\" (UID: \"76b48563-d38b-4e30-af57-c37cc7b66a9e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-642k8"
Oct 30 22:08:23.486964 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.486944    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mkwp\" (UniqueName: \"kubernetes.io/projected/ba0fc359-3a19-49ea-b798-2ca0080b30b9-kube-api-access-8mkwp\") pod \"kas-connection-checker-685895c7c9-79956\" (UID: \"ba0fc359-3a19-49ea-b798-2ca0080b30b9\") " pod="kube-system/kas-connection-checker-685895c7c9-79956"
Oct 30 22:08:23.487134 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.487119    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-bound-sa-token\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:23.487230 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.487212    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgr7f\" (UniqueName: \"kubernetes.io/projected/41dcf151-1d72-4978-b0aa-2046f213cb7e-kube-api-access-rgr7f\") pod \"node-ca-bc87g\" (UID: \"41dcf151-1d72-4978-b0aa-2046f213cb7e\") " pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:08:23.487321 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.487308    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c4bd\" (UniqueName: \"kubernetes.io/projected/3788135a-d420-4738-a292-845c50e486a7-kube-api-access-6c4bd\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:23.488134 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.488117    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx7tp\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-kube-api-access-fx7tp\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:23.488654 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.488639    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vppkm\" (UniqueName: \"kubernetes.io/projected/94490976-1f2d-4db5-980a-322cad7c057b-kube-api-access-vppkm\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:23.489173 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.489154    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkwmz\" (UniqueName: \"kubernetes.io/projected/d81c8448-4580-4174-a7d8-28f5aadf238f-kube-api-access-vkwmz\") pod \"ingress-canary-svwl7\" (UID: \"d81c8448-4580-4174-a7d8-28f5aadf238f\") " pod="openshift-ingress-canary/ingress-canary-svwl7"
Oct 30 22:08:23.489230 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.489216    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4pmg\" (UniqueName: \"kubernetes.io/projected/557fc88c-2102-45ec-adc4-80bae4f140cb-kube-api-access-h4pmg\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:23.490403 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.490382    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4c8\" (UniqueName: \"kubernetes.io/projected/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-api-access-jz4c8\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:23.490441 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.490408    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8kpl\" (UniqueName: \"kubernetes.io/projected/09b48430-dd49-439d-b317-384408fec509-kube-api-access-f8kpl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp\" (UID: \"09b48430-dd49-439d-b317-384408fec509\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp"
Oct 30 22:08:23.490614 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.490600    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwmpt\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-kube-api-access-rwmpt\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:23.526513 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526494    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-var-lib-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.526564 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526519    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-log-socket\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.526564 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526533    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-run-ovn-kubernetes\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.526564 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526548    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:23.526706 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526572    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-etc-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.526706 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526587    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-log-socket\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.526706 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526598    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:23.526706 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526610    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-run-ovn-kubernetes\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.526706 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526631    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-var-lib-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.526706 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526633    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-kubelet\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.526706 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526666    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-kubelet\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.526706 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526635    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-etc-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.526706 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526683    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysctl-d\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.526710    4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.526721    4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdbcb7997-ggr5g: secret "image-registry-tls" not found
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.526725    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:24.02670656 +0000 UTC m=+2.026054125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : configmap references non-existent config key: service-ca.crt
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526745    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-sys-fs\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.526772    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls podName:8969718f-1dae-4bde-81df-592ee58a1f76 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:24.026756442 +0000 UTC m=+2.026104005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls") pod "image-registry-bdbcb7997-ggr5g" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76") : secret "image-registry-tls" not found
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526787    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-run\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526798    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-sys-fs\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526806    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526798    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysctl-d\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526823    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-ovn\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526836    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-run\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526846    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-system-cni-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526875    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-device-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526890    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-kubelet\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.526904    4947 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526931    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-system-cni-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526906    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526953    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-ovn\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.526964    4947 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526981    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-device-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.526986    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-kubelet\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.526968    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert podName:ba52011d-6892-4b4d-ac14-07e8be859d9e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:24.026953036 +0000 UTC m=+2.026300589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8sk5t" (UID: "ba52011d-6892-4b4d-ac14-07e8be859d9e") : secret "networking-console-plugin-cert" not found
Oct 30 22:08:23.527034 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.527040    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs podName:3788135a-d420-4738-a292-845c50e486a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:24.027022965 +0000 UTC m=+2.026370509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs") pod "network-metrics-daemon-g5pws" (UID: "3788135a-d420-4738-a292-845c50e486a7") : secret "metrics-daemon-secret" not found
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527057    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93f981a6-4af6-4405-aec6-89e539c42c67-dbus\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527074    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-cnibin\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527105    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-cnibin\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527115    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93f981a6-4af6-4405-aec6-89e539c42c67-dbus\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527116    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-netns\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527147    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-hostroot\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527175    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527192    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-netns\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527202    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527215    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-hostroot\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527222    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527228    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-hosts-file\") pod \"node-resolver-s2528\" (UID: \"f92ebb16-2530-4f59-98d6-c3c2695ad9ba\") " pod="openshift-dns/node-resolver-s2528"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527251    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-kubernetes\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.527267    4947 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527276    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f92ebb16-2530-4f59-98d6-c3c2695ad9ba-hosts-file\") pod \"node-resolver-s2528\" (UID: \"f92ebb16-2530-4f59-98d6-c3c2695ad9ba\") " pod="openshift-dns/node-resolver-s2528"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527276    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-registration-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.527302    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls podName:557fc88c-2102-45ec-adc4-80bae4f140cb nodeName:}" failed. No retries permitted until 2026-10-30 22:08:24.027291832 +0000 UTC m=+2.026639372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-k5qrx" (UID: "557fc88c-2102-45ec-adc4-80bae4f140cb") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527305    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-kubernetes\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527313    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527347    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-cni-bin\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527359    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-registration-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527364    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-cni-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527392    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527400    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-cni-bin\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527405    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-cni-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527401    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-host\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527444    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-slash\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527468    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-systemd\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527494    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-host\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527493    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdae7310-568d-48ec-80e7-841bd98ab04e-crio-socket\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527522    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-systemd\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527526    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-var-lib-kubelet\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527496    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-slash\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527536    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdae7310-568d-48ec-80e7-841bd98ab04e-crio-socket\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527546    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93f981a6-4af6-4405-aec6-89e539c42c67-kubelet-config\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527574    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-multus-certs\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527584    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93f981a6-4af6-4405-aec6-89e539c42c67-kubelet-config\") pod \"global-pull-secret-syncer-xtgwp\" (UID: \"93f981a6-4af6-4405-aec6-89e539c42c67\") " pod="kube-system/global-pull-secret-syncer-xtgwp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527589    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-var-lib-kubelet\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527596    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysconfig\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527621    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527631    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-multus-certs\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527624    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysconfig\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527650    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-lib-modules\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527671    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-cni-netd\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527706    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-systemd-units\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527731    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527752    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-sys\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527772    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-run-netns\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527771    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-cni-netd\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527789    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-systemd-units\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527810    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-lib-modules\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527816    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-sys\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527827    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-host-run-netns\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527852    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.527867    4947 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527888    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-modprobe-d\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527916    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-systemd\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.527948    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls podName:fdae7310-568d-48ec-80e7-841bd98ab04e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:24.027938643 +0000 UTC m=+2.027286183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqpbw" (UID: "fdae7310-568d-48ec-80e7-841bd98ab04e") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.527968    4947 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528003    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-cnibin\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528002    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-modprobe-d\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527971    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-systemd\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.527967    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-cnibin\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.528043    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls podName:610534ce-c054-41fd-8ab6-b70696b7e360 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:24.02802908 +0000 UTC m=+2.027376623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-v6j52" (UID: "610534ce-c054-41fd-8ab6-b70696b7e360") : secret "samples-operator-tls" not found
Oct 30 22:08:23.528014 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528058    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdae7310-568d-48ec-80e7-841bd98ab04e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528077    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-k8s-cni-cncf-io\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528092    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-cni-multus\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528107    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-node-log\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528124    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-socket-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528159    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-cni-multus\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528171    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-node-log\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528177    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-socket-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528188    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-conf-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528222    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528182    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-run-k8s-cni-cncf-io\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528248    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-cni-bin\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528257    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-conf-dir\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528274    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-host-var-lib-cni-bin\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528291    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.528311    4947 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528317    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.528320    4947 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.528362    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls podName:a4369922-ceac-4aaf-9a8b-35c5a0f27d91 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:24.028348969 +0000 UTC m=+2.027696510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls") pod "dns-default-jg2dx" (UID: "a4369922-ceac-4aaf-9a8b-35c5a0f27d91") : secret "dns-default-metrics-tls" not found
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528366    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79823d82-b501-47c9-acae-667e192d721b-run-openvswitch\") pod \"ovnkube-node-z2zrp\" (UID: \"79823d82-b501-47c9-acae-667e192d721b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528376    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.528390    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:24.028378051 +0000 UTC m=+2.027725612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : secret "router-metrics-certs-default" not found
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528417    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528445    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7048b86e-5ca2-46c8-8b2d-79116829faf2-host-slash\") pod \"iptables-alerter-c5whj\" (UID: \"7048b86e-5ca2-46c8-8b2d-79116829faf2\") " pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528452    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528469    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-etc-selinux\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.528422    4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.528493    4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ff86975d4-6xkd8: secret "image-registry-tls" not found
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528495    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-system-cni-dir\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528517    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7048b86e-5ca2-46c8-8b2d-79116829faf2-host-slash\") pod \"iptables-alerter-c5whj\" (UID: \"7048b86e-5ca2-46c8-8b2d-79116829faf2\") " pod="openshift-network-operator/iptables-alerter-c5whj"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:23.528527    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls podName:4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d nodeName:}" failed. No retries permitted until 2026-10-30 22:08:24.028518501 +0000 UTC m=+2.027866042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls") pod "image-registry-5ff86975d4-6xkd8" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d") : secret "image-registry-tls" not found
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528542    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bec12d9c-76a5-463a-991e-ca82c18fccd2-system-cni-dir\") pod \"multus-additional-cni-plugins-cs9rl\" (UID: \"bec12d9c-76a5-463a-991e-ca82c18fccd2\") " pod="openshift-multus/multus-additional-cni-plugins-cs9rl"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528544    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac726c56-0d31-4b0d-ae06-e547b7ba1dbd-etc-selinux\") pod \"aws-ebs-csi-driver-node-j2jbl\" (UID: \"ac726c56-0d31-4b0d-ae06-e547b7ba1dbd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2jbl"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528582    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41dcf151-1d72-4978-b0aa-2046f213cb7e-host\") pod \"node-ca-bc87g\" (UID: \"41dcf151-1d72-4978-b0aa-2046f213cb7e\") " pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528600    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-socket-dir-parent\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528614    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-etc-kubernetes\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528618    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41dcf151-1d72-4978-b0aa-2046f213cb7e-host\") pod \"node-ca-bc87g\" (UID: \"41dcf151-1d72-4978-b0aa-2046f213cb7e\") " pod="openshift-image-registry/node-ca-bc87g"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528629    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysctl-conf\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528655    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-etc-kubernetes\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528656    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4d3f83c-7808-4a9a-84d1-d9428d5bd592-multus-socket-dir-parent\") pod \"multus-2ptq8\" (UID: \"b4d3f83c-7808-4a9a-84d1-d9428d5bd592\") " pod="openshift-multus/multus-2ptq8"
Oct 30 22:08:23.529970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.528724    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0a30486-a1f5-4404-8303-6efed2dad370-etc-sysctl-conf\") pod \"tuned-np6dm\" (UID: \"c0a30486-a1f5-4404-8303-6efed2dad370\") " pod="openshift-cluster-node-tuning-operator/tuned-np6dm"
Oct 30 22:08:23.554193 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.554170    4947 scope.go:122] "RemoveContainer" containerID="bab56f6146fa081692f161b3f11e5251a5f21f023755d1e5896cb70b10996697"
Oct 30 22:08:23.570802 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:23.570782    4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:08:24.032576 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:24.032498    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:24.032576 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:24.032531    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:24.032576 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:24.032552    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:24.032576 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:24.032579    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:24.032936 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:24.032597    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:24.032936 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.032656    4947 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:08:24.032936 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.032669    4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:24.032936 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.032682    4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdbcb7997-ggr5g: secret "image-registry-tls" not found
Oct 30 22:08:24.032936 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:24.032688    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:24.032936 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.032664    4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:24.032936 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.032722    4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ff86975d4-6xkd8: secret "image-registry-tls" not found
Oct 30 22:08:24.032936 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.032723    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:25.032703197 +0000 UTC m=+3.032050756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : configmap references non-existent config key: service-ca.crt
Oct 30 22:08:24.032936 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.032659    4947 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:08:24.032936 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.032741    4947 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:08:24.032936 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.032763    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls podName:4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d nodeName:}" failed. No retries permitted until 2026-10-30 22:08:25.032747589 +0000 UTC m=+3.032095130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls") pod "image-registry-5ff86975d4-6xkd8" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d") : secret "image-registry-tls" not found
Oct 30 22:08:24.032936 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.032788    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls podName:a4369922-ceac-4aaf-9a8b-35c5a0f27d91 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:25.032777827 +0000 UTC m=+3.032125373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls") pod "dns-default-jg2dx" (UID: "a4369922-ceac-4aaf-9a8b-35c5a0f27d91") : secret "dns-default-metrics-tls" not found
Oct 30 22:08:24.032936 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.032803    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls podName:8969718f-1dae-4bde-81df-592ee58a1f76 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:25.03279763 +0000 UTC m=+3.032145171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls") pod "image-registry-bdbcb7997-ggr5g" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76") : secret "image-registry-tls" not found
Oct 30 22:08:24.032936 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.032815    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:25.03280944 +0000 UTC m=+3.032156981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : secret "router-metrics-certs-default" not found
Oct 30 22:08:24.032936 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.032825    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert podName:ba52011d-6892-4b4d-ac14-07e8be859d9e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:25.032820533 +0000 UTC m=+3.032168073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8sk5t" (UID: "ba52011d-6892-4b4d-ac14-07e8be859d9e") : secret "networking-console-plugin-cert" not found
Oct 30 22:08:24.032936 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:24.032865    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:24.032936 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:24.032894    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:24.033565 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:24.032945    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:24.033565 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:24.032977    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:24.033565 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.032980    4947 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:08:24.033565 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.033024    4947 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:24.033565 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.033040    4947 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:08:24.033565 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.033056    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs podName:3788135a-d420-4738-a292-845c50e486a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:25.033046293 +0000 UTC m=+3.032393836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs") pod "network-metrics-daemon-g5pws" (UID: "3788135a-d420-4738-a292-845c50e486a7") : secret "metrics-daemon-secret" not found
Oct 30 22:08:24.033565 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.033072    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls podName:fdae7310-568d-48ec-80e7-841bd98ab04e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:25.033063512 +0000 UTC m=+3.032411053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqpbw" (UID: "fdae7310-568d-48ec-80e7-841bd98ab04e") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:08:24.033565 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.033080    4947 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:08:24.033565 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.033086    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls podName:557fc88c-2102-45ec-adc4-80bae4f140cb nodeName:}" failed. No retries permitted until 2026-10-30 22:08:25.033079923 +0000 UTC m=+3.032427463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-k5qrx" (UID: "557fc88c-2102-45ec-adc4-80bae4f140cb") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:24.033565 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.033126    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls podName:610534ce-c054-41fd-8ab6-b70696b7e360 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:25.0331117 +0000 UTC m=+3.032459244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-v6j52" (UID: "610534ce-c054-41fd-8ab6-b70696b7e360") : secret "samples-operator-tls" not found
Oct 30 22:08:24.209248 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:24.209220    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/2.log"
Oct 30 22:08:24.209664 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:24.209617    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/1.log"
Oct 30 22:08:24.209664 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:24.209649    4947 generic.go:356] "Generic (PLEG): container finished" podID="00c23fb6-cbfb-4498-ad17-15998e6d2d66" containerID="d952c886b3a91643d91088cd9faaeec9cdfa6b3f55d97b201c1c656fb70f7c48" exitCode=255
Oct 30 22:08:24.209751 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:24.209731    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr" event={"ID":"00c23fb6-cbfb-4498-ad17-15998e6d2d66","Type":"ContainerDied","Data":"d952c886b3a91643d91088cd9faaeec9cdfa6b3f55d97b201c1c656fb70f7c48"}
Oct 30 22:08:24.209806 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:24.209792    4947 scope.go:122] "RemoveContainer" containerID="bab56f6146fa081692f161b3f11e5251a5f21f023755d1e5896cb70b10996697"
Oct 30 22:08:24.210197 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:24.210176    4947 scope.go:122] "RemoveContainer" containerID="d952c886b3a91643d91088cd9faaeec9cdfa6b3f55d97b201c1c656fb70f7c48"
Oct 30 22:08:24.210358 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:24.210314    4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gs4xr_openshift-console-operator(00c23fb6-cbfb-4498-ad17-15998e6d2d66)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr" podUID="00c23fb6-cbfb-4498-ad17-15998e6d2d66"
Oct 30 22:08:24.825452 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:24.825423    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:08:25.040621 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:25.040587    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:25.040801 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:25.040628    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:25.040801 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:25.040676    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:25.040801 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:25.040694    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:25.040801 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:25.040723    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:25.040801 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:25.040741    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:25.040801 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.040763    4947 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:08:25.041050 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.040822    4947 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:08:25.041050 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.040827    4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:25.041050 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.040763    4947 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:08:25.041050 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.040826    4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:25.041050 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.040882    4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ff86975d4-6xkd8: secret "image-registry-tls" not found
Oct 30 22:08:25.041050 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.040834    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls podName:610534ce-c054-41fd-8ab6-b70696b7e360 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:27.040814223 +0000 UTC m=+5.040161782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-v6j52" (UID: "610534ce-c054-41fd-8ab6-b70696b7e360") : secret "samples-operator-tls" not found
Oct 30 22:08:25.041050 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.040844    4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdbcb7997-ggr5g: secret "image-registry-tls" not found
Oct 30 22:08:25.041050 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:25.040921    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:25.041050 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.040937    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls podName:8969718f-1dae-4bde-81df-592ee58a1f76 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:27.040929845 +0000 UTC m=+5.040277386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls") pod "image-registry-bdbcb7997-ggr5g" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76") : secret "image-registry-tls" not found
Oct 30 22:08:25.041050 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.040951    4947 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:08:25.041050 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.040954    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:27.040949095 +0000 UTC m=+5.040296636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : secret "router-metrics-certs-default" not found
Oct 30 22:08:25.041050 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.040969    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:27.040962029 +0000 UTC m=+5.040309571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : configmap references non-existent config key: service-ca.crt
Oct 30 22:08:25.041050 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.040981    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls podName:a4369922-ceac-4aaf-9a8b-35c5a0f27d91 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:27.040976318 +0000 UTC m=+5.040323858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls") pod "dns-default-jg2dx" (UID: "a4369922-ceac-4aaf-9a8b-35c5a0f27d91") : secret "dns-default-metrics-tls" not found
Oct 30 22:08:25.041050 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.041002    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls podName:4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d nodeName:}" failed. No retries permitted until 2026-10-30 22:08:27.040987102 +0000 UTC m=+5.040334643 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls") pod "image-registry-5ff86975d4-6xkd8" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d") : secret "image-registry-tls" not found
Oct 30 22:08:25.041050 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.041019    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert podName:ba52011d-6892-4b4d-ac14-07e8be859d9e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:27.041013753 +0000 UTC m=+5.040361295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8sk5t" (UID: "ba52011d-6892-4b4d-ac14-07e8be859d9e") : secret "networking-console-plugin-cert" not found
Oct 30 22:08:25.041050 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:25.041052    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:25.041622 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:25.041080    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:25.041622 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:25.041122    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:25.041622 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.041163    4947 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:08:25.041622 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.041182    4947 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:08:25.041622 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.041194    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs podName:3788135a-d420-4738-a292-845c50e486a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:27.04118128 +0000 UTC m=+5.040528821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs") pod "network-metrics-daemon-g5pws" (UID: "3788135a-d420-4738-a292-845c50e486a7") : secret "metrics-daemon-secret" not found
Oct 30 22:08:25.041622 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.041206    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls podName:fdae7310-568d-48ec-80e7-841bd98ab04e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:27.041199785 +0000 UTC m=+5.040547372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqpbw" (UID: "fdae7310-568d-48ec-80e7-841bd98ab04e") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:08:25.041622 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.041212    4947 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:25.041622 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:25.041275    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls podName:557fc88c-2102-45ec-adc4-80bae4f140cb nodeName:}" failed. No retries permitted until 2026-10-30 22:08:27.041258291 +0000 UTC m=+5.040605879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-k5qrx" (UID: "557fc88c-2102-45ec-adc4-80bae4f140cb") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:25.214878 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:25.214790    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-642k8" event={"ID":"76b48563-d38b-4e30-af57-c37cc7b66a9e","Type":"ContainerStarted","Data":"2f344d4eba4699f456c7090e84cf4c1ec7d7d4708d61aab10129c64196c715b6"}
Oct 30 22:08:25.216590 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:25.216571    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/2.log"
Oct 30 22:08:26.414386 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:26.414354    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:26.430742 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:26.430723    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s2528_f92ebb16-2530-4f59-98d6-c3c2695ad9ba/dns-node-resolver/0.log"
Oct 30 22:08:26.872805 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:26.872768    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:26.873102 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:26.873089    4947 scope.go:122] "RemoveContainer" containerID="d952c886b3a91643d91088cd9faaeec9cdfa6b3f55d97b201c1c656fb70f7c48"
Oct 30 22:08:26.873265 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:26.873248    4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gs4xr_openshift-console-operator(00c23fb6-cbfb-4498-ad17-15998e6d2d66)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr" podUID="00c23fb6-cbfb-4498-ad17-15998e6d2d66"
Oct 30 22:08:26.957535 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:26.957504    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bc87g_41dcf151-1d72-4978-b0aa-2046f213cb7e/node-ca/0.log"
Oct 30 22:08:27.058234 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:27.058189    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:27.058234 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:27.058238    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:27.058474 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:27.058262    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:27.058474 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:27.058290    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:27.058474 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:27.058321    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:27.058474 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058360    4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:27.058474 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058381    4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdbcb7997-ggr5g: secret "image-registry-tls" not found
Oct 30 22:08:27.058474 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058409    4947 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:08:27.058474 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058381    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:31.058361654 +0000 UTC m=+9.057709209 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : configmap references non-existent config key: service-ca.crt
Oct 30 22:08:27.058474 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058423    4947 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:27.058474 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058455    4947 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:08:27.058474 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058470    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls podName:8969718f-1dae-4bde-81df-592ee58a1f76 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:31.058436415 +0000 UTC m=+9.057783964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls") pod "image-registry-bdbcb7997-ggr5g" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76") : secret "image-registry-tls" not found
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058491    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert podName:ba52011d-6892-4b4d-ac14-07e8be859d9e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:31.058482911 +0000 UTC m=+9.057830451 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8sk5t" (UID: "ba52011d-6892-4b4d-ac14-07e8be859d9e") : secret "networking-console-plugin-cert" not found
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058503    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls podName:557fc88c-2102-45ec-adc4-80bae4f140cb nodeName:}" failed. No retries permitted until 2026-10-30 22:08:31.058497424 +0000 UTC m=+9.057844965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-k5qrx" (UID: "557fc88c-2102-45ec-adc4-80bae4f140cb") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058514    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs podName:3788135a-d420-4738-a292-845c50e486a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:31.058508969 +0000 UTC m=+9.057856511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs") pod "network-metrics-daemon-g5pws" (UID: "3788135a-d420-4738-a292-845c50e486a7") : secret "metrics-daemon-secret" not found
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:27.058542    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:27.058566    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:27.058592    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:27.058610    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:27.058637    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058656    4947 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058676    4947 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058685    4947 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058696    4947 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058714    4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058726    4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ff86975d4-6xkd8: secret "image-registry-tls" not found
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058700    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls podName:fdae7310-568d-48ec-80e7-841bd98ab04e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:31.058687042 +0000 UTC m=+9.058034586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqpbw" (UID: "fdae7310-568d-48ec-80e7-841bd98ab04e") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058754    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls podName:610534ce-c054-41fd-8ab6-b70696b7e360 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:31.058738985 +0000 UTC m=+9.058086532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-v6j52" (UID: "610534ce-c054-41fd-8ab6-b70696b7e360") : secret "samples-operator-tls" not found
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058768    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls podName:a4369922-ceac-4aaf-9a8b-35c5a0f27d91 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:31.058761588 +0000 UTC m=+9.058109129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls") pod "dns-default-jg2dx" (UID: "a4369922-ceac-4aaf-9a8b-35c5a0f27d91") : secret "dns-default-metrics-tls" not found
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058777    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:31.058772615 +0000 UTC m=+9.058120156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : secret "router-metrics-certs-default" not found
Oct 30 22:08:27.058877 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:27.058789    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls podName:4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d nodeName:}" failed. No retries permitted until 2026-10-30 22:08:31.058785198 +0000 UTC m=+9.058132738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls") pod "image-registry-5ff86975d4-6xkd8" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d") : secret "image-registry-tls" not found
Oct 30 22:08:27.954177 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:27.954145    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-svwl7_d81c8448-4580-4174-a7d8-28f5aadf238f/serve-healthcheck-canary/0.log"
Oct 30 22:08:28.353907 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:28.353878    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-642k8_76b48563-d38b-4e30-af57-c37cc7b66a9e/migrator/0.log"
Oct 30 22:08:28.554700 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:28.554662    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp_09b48430-dd49-439d-b317-384408fec509/kube-storage-version-migrator-operator/0.log"
Oct 30 22:08:28.642438 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:28.642323    4947 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:28.642779 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:28.642766    4947 scope.go:122] "RemoveContainer" containerID="d952c886b3a91643d91088cd9faaeec9cdfa6b3f55d97b201c1c656fb70f7c48"
Oct 30 22:08:28.642959 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:28.642940    4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gs4xr_openshift-console-operator(00c23fb6-cbfb-4498-ad17-15998e6d2d66)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr" podUID="00c23fb6-cbfb-4498-ad17-15998e6d2d66"
Oct 30 22:08:29.275171 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:29.275143    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:29.275664 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:29.275305    4947 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:08:31.098684 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.098650    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.098689    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.098716    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.098735    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.098753    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.098782    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.098798    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.098819    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.098849    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.098885    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:31.098904    4947 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:31.098921    4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:31.098942    4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:31.098944    4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ff86975d4-6xkd8: secret "image-registry-tls" not found
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:31.098959    4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdbcb7997-ggr5g: secret "image-registry-tls" not found
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:31.098971    4947 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:31.098983    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls podName:a4369922-ceac-4aaf-9a8b-35c5a0f27d91 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:39.098962761 +0000 UTC m=+17.098310321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls") pod "dns-default-jg2dx" (UID: "a4369922-ceac-4aaf-9a8b-35c5a0f27d91") : secret "dns-default-metrics-tls" not found
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:31.098904    4947 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:31.099022    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls podName:4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d nodeName:}" failed. No retries permitted until 2026-10-30 22:08:39.099004593 +0000 UTC m=+17.098352147 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls") pod "image-registry-5ff86975d4-6xkd8" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d") : secret "image-registry-tls" not found
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:31.099040    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls podName:8969718f-1dae-4bde-81df-592ee58a1f76 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:39.099031101 +0000 UTC m=+17.098378642 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls") pod "image-registry-bdbcb7997-ggr5g" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76") : secret "image-registry-tls" not found
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:31.099056    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert podName:ba52011d-6892-4b4d-ac14-07e8be859d9e nodeName:}" failed. No retries permitted until 2026-10-30 22:08:39.099046827 +0000 UTC m=+17.098394391 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-8sk5t" (UID: "ba52011d-6892-4b4d-ac14-07e8be859d9e") : secret "networking-console-plugin-cert" not found
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:31.099070    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:39.099062909 +0000 UTC m=+17.098410453 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : secret "router-metrics-certs-default" not found
Oct 30 22:08:31.099119 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.099123    4947 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 22:08:31.099905 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:31.099182    4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle podName:94490976-1f2d-4db5-980a-322cad7c057b nodeName:}" failed. No retries permitted until 2026-10-30 22:08:39.09916929 +0000 UTC m=+17.098516831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle") pod "router-default-56d5748b65-v9kvx" (UID: "94490976-1f2d-4db5-980a-322cad7c057b") : configmap references non-existent config key: service-ca.crt
Oct 30 22:08:31.101355 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.101305    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdae7310-568d-48ec-80e7-841bd98ab04e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqpbw\" (UID: \"fdae7310-568d-48ec-80e7-841bd98ab04e\") " pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:31.101544 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.101527    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/557fc88c-2102-45ec-adc4-80bae4f140cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-k5qrx\" (UID: \"557fc88c-2102-45ec-adc4-80bae4f140cb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:31.101827 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.101810    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3788135a-d420-4738-a292-845c50e486a7-metrics-certs\") pod \"network-metrics-daemon-g5pws\" (UID: \"3788135a-d420-4738-a292-845c50e486a7\") " pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:31.101923 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.101905    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/610534ce-c054-41fd-8ab6-b70696b7e360-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-v6j52\" (UID: \"610534ce-c054-41fd-8ab6-b70696b7e360\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:31.353825 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.353735    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"
Oct 30 22:08:31.353825 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.353778    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"
Oct 30 22:08:31.363918 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.363898    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g5pws"
Oct 30 22:08:31.366510 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.366491    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dqpbw"
Oct 30 22:08:31.740090 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.739999    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx"]
Oct 30 22:08:31.776774 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:31.776739    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557fc88c_2102_45ec_adc4_80bae4f140cb.slice/crio-b0778bc2033027f922f4be09b188816d7cf107413a187e3a1aa744a5166f98e1 WatchSource:0}: Error finding container b0778bc2033027f922f4be09b188816d7cf107413a187e3a1aa744a5166f98e1: Status 404 returned error can't find the container with id b0778bc2033027f922f4be09b188816d7cf107413a187e3a1aa744a5166f98e1
Oct 30 22:08:31.785586 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.785563    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g5pws"]
Oct 30 22:08:31.788655 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:31.788627    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3788135a_d420_4738_a292_845c50e486a7.slice/crio-2aa1048e3b118c07da35203c2b44faae032a828d50ec54c2fd5613de157094e5 WatchSource:0}: Error finding container 2aa1048e3b118c07da35203c2b44faae032a828d50ec54c2fd5613de157094e5: Status 404 returned error can't find the container with id 2aa1048e3b118c07da35203c2b44faae032a828d50ec54c2fd5613de157094e5
Oct 30 22:08:31.789690 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.789673    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52"]
Oct 30 22:08:31.791237 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:31.791163    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dqpbw"]
Oct 30 22:08:31.827830 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:31.827795    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdae7310_568d_48ec_80e7_841bd98ab04e.slice/crio-2ba7812f869a3acc7727e29e77bf52bf70af07043a3788927c564cb964e963e3 WatchSource:0}: Error finding container 2ba7812f869a3acc7727e29e77bf52bf70af07043a3788927c564cb964e963e3: Status 404 returned error can't find the container with id 2ba7812f869a3acc7727e29e77bf52bf70af07043a3788927c564cb964e963e3
Oct 30 22:08:32.240630 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:32.240597    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52" event={"ID":"610534ce-c054-41fd-8ab6-b70696b7e360","Type":"ContainerStarted","Data":"27dbd07f05de1e7a9b5ecd317648a5d15d37c4c48f22a2ac93bb700565ada68a"}
Oct 30 22:08:32.241744 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:32.241719    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g5pws" event={"ID":"3788135a-d420-4738-a292-845c50e486a7","Type":"ContainerStarted","Data":"2aa1048e3b118c07da35203c2b44faae032a828d50ec54c2fd5613de157094e5"}
Oct 30 22:08:32.242899 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:32.242875    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx" event={"ID":"557fc88c-2102-45ec-adc4-80bae4f140cb","Type":"ContainerStarted","Data":"b0778bc2033027f922f4be09b188816d7cf107413a187e3a1aa744a5166f98e1"}
Oct 30 22:08:32.244569 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:32.244546    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dqpbw" event={"ID":"fdae7310-568d-48ec-80e7-841bd98ab04e","Type":"ContainerStarted","Data":"fd0eeb2a3bbdfa23a3a1b0d03923c9610c183862101d208e4dee8049c015519c"}
Oct 30 22:08:32.244692 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:32.244589    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dqpbw" event={"ID":"fdae7310-568d-48ec-80e7-841bd98ab04e","Type":"ContainerStarted","Data":"2ba7812f869a3acc7727e29e77bf52bf70af07043a3788927c564cb964e963e3"}
Oct 30 22:08:33.251319 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:33.251228    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dqpbw" event={"ID":"fdae7310-568d-48ec-80e7-841bd98ab04e","Type":"ContainerStarted","Data":"f83b255ce2edeb24e9f2c7549be260321ba9459690cd7b19d009ad52a91b39c7"}
Oct 30 22:08:34.257415 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:34.257382    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g5pws" event={"ID":"3788135a-d420-4738-a292-845c50e486a7","Type":"ContainerStarted","Data":"d03b86878722fc44395f6bc382778515a825e9bc84326c5371d0a7c7c1ce25fe"}
Oct 30 22:08:34.257812 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:34.257448    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g5pws" event={"ID":"3788135a-d420-4738-a292-845c50e486a7","Type":"ContainerStarted","Data":"1b15ba66af32fedb6a7f8c0d040b16c20662db1f8cf0ec23d59d43d0bc734b6f"}
Oct 30 22:08:35.262506 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:35.262425    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dqpbw" event={"ID":"fdae7310-568d-48ec-80e7-841bd98ab04e","Type":"ContainerStarted","Data":"45e0a39db93669012fd039a2eec2144f97b65beb26f2576ba705856c8b9beb44"}
Oct 30 22:08:35.264270 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:35.264241    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52" event={"ID":"610534ce-c054-41fd-8ab6-b70696b7e360","Type":"ContainerStarted","Data":"8b7158f008aa0fd719e20a3c9adbf8e41064d918c16531e67f5ac055591fce5b"}
Oct 30 22:08:35.264416 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:35.264303    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-v6j52" event={"ID":"610534ce-c054-41fd-8ab6-b70696b7e360","Type":"ContainerStarted","Data":"2b0d209fa65240196fb06c0d26d33936218a0c01e8dfdb6f49a47e3c7035d378"}
Oct 30 22:08:35.265710 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:35.265685    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-k5qrx" event={"ID":"557fc88c-2102-45ec-adc4-80bae4f140cb","Type":"ContainerStarted","Data":"20598457aaba20941040ca2fb2448423cafa02e141107e215f319ef261275258"}
Oct 30 22:08:36.152949 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:36.152918    4947 scope.go:122] "RemoveContainer" containerID="d952c886b3a91643d91088cd9faaeec9cdfa6b3f55d97b201c1c656fb70f7c48"
Oct 30 22:08:36.270582 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:36.270559    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:08:36.270996 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:36.270980    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/2.log"
Oct 30 22:08:36.271038 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:36.271016    4947 generic.go:356] "Generic (PLEG): container finished" podID="00c23fb6-cbfb-4498-ad17-15998e6d2d66" containerID="3d92772ec7039ec63e1a12df8bcd2e87eabf32e5b9cd00298233567ca6a6f4cf" exitCode=255
Oct 30 22:08:36.271158 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:36.271131    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr" event={"ID":"00c23fb6-cbfb-4498-ad17-15998e6d2d66","Type":"ContainerDied","Data":"3d92772ec7039ec63e1a12df8bcd2e87eabf32e5b9cd00298233567ca6a6f4cf"}
Oct 30 22:08:36.271228 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:36.271198    4947 scope.go:122] "RemoveContainer" containerID="d952c886b3a91643d91088cd9faaeec9cdfa6b3f55d97b201c1c656fb70f7c48"
Oct 30 22:08:36.271817 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:36.271788    4947 scope.go:122] "RemoveContainer" containerID="3d92772ec7039ec63e1a12df8bcd2e87eabf32e5b9cd00298233567ca6a6f4cf"
Oct 30 22:08:36.271987 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:36.271965    4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gs4xr_openshift-console-operator(00c23fb6-cbfb-4498-ad17-15998e6d2d66)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr" podUID="00c23fb6-cbfb-4498-ad17-15998e6d2d66"
Oct 30 22:08:36.873613 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:36.873580    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:36.991430 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:36.991405    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z2zrp"
Oct 30 22:08:37.276174 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:37.276090    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:08:37.276613 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:37.276476    4947 scope.go:122] "RemoveContainer" containerID="3d92772ec7039ec63e1a12df8bcd2e87eabf32e5b9cd00298233567ca6a6f4cf"
Oct 30 22:08:37.276690 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:37.276670    4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gs4xr_openshift-console-operator(00c23fb6-cbfb-4498-ad17-15998e6d2d66)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr" podUID="00c23fb6-cbfb-4498-ad17-15998e6d2d66"
Oct 30 22:08:38.642797 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:38.642763    4947 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:08:38.643224 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:38.643128    4947 scope.go:122] "RemoveContainer" containerID="3d92772ec7039ec63e1a12df8bcd2e87eabf32e5b9cd00298233567ca6a6f4cf"
Oct 30 22:08:38.643371 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:38.643326    4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gs4xr_openshift-console-operator(00c23fb6-cbfb-4498-ad17-15998e6d2d66)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr" podUID="00c23fb6-cbfb-4498-ad17-15998e6d2d66"
Oct 30 22:08:38.723018 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:38.722985    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fgkn6"
Oct 30 22:08:39.178460 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.178422    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:39.178460 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.178460    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:39.178730 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.178480    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:39.178730 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.178613    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:39.178730 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.178660    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:39.178730 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.178691    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:39.179614 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.179584    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94490976-1f2d-4db5-980a-322cad7c057b-service-ca-bundle\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:39.181119 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.181096    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4369922-ceac-4aaf-9a8b-35c5a0f27d91-metrics-tls\") pod \"dns-default-jg2dx\" (UID: \"a4369922-ceac-4aaf-9a8b-35c5a0f27d91\") " pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:39.181262 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.181242    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba52011d-6892-4b4d-ac14-07e8be859d9e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-8sk5t\" (UID: \"ba52011d-6892-4b4d-ac14-07e8be859d9e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:39.181374 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.181290    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls\") pod \"image-registry-bdbcb7997-ggr5g\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") " pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:39.181590 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.181573    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94490976-1f2d-4db5-980a-322cad7c057b-metrics-certs\") pod \"router-default-56d5748b65-v9kvx\" (UID: \"94490976-1f2d-4db5-980a-322cad7c057b\") " pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:39.181674 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.181653    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls\") pod \"image-registry-5ff86975d4-6xkd8\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") " pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:39.458265 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.458175    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:39.459246 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.459225    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:39.459305 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.459289    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"
Oct 30 22:08:39.465322 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.465305    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:39.465428 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.465415    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:39.960397 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:39.960354    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-8sk5t"]
Oct 30 22:08:39.963848 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:39.963799    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba52011d_6892_4b4d_ac14_07e8be859d9e.slice/crio-ab004c5e707c7bea77af66a1fa13104d752171dfc8c92dbca950cf997c7f1af8 WatchSource:0}: Error finding container ab004c5e707c7bea77af66a1fa13104d752171dfc8c92dbca950cf997c7f1af8: Status 404 returned error can't find the container with id ab004c5e707c7bea77af66a1fa13104d752171dfc8c92dbca950cf997c7f1af8
Oct 30 22:08:39.968259 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:39.968237    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8969718f_1dae_4bde_81df_592ee58a1f76.slice/crio-d2ef17f005a8aebde665278f4ffc8f7da12a76d9fa7a9fa8c87edc0f7909b56b WatchSource:0}: Error finding container d2ef17f005a8aebde665278f4ffc8f7da12a76d9fa7a9fa8c87edc0f7909b56b: Status 404 returned error can't find the container with id d2ef17f005a8aebde665278f4ffc8f7da12a76d9fa7a9fa8c87edc0f7909b56b
Oct 30 22:08:39.995469 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:39.995442    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94490976_1f2d_4db5_980a_322cad7c057b.slice/crio-24fdcc6041a820a73cf95097fa5cf1ca1b6296c5b18c0ae639ca5c7b30acd9de WatchSource:0}: Error finding container 24fdcc6041a820a73cf95097fa5cf1ca1b6296c5b18c0ae639ca5c7b30acd9de: Status 404 returned error can't find the container with id 24fdcc6041a820a73cf95097fa5cf1ca1b6296c5b18c0ae639ca5c7b30acd9de
Oct 30 22:08:39.995714 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:39.995682    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0035aa_74bd_417b_ba1e_0ebcb58c6e3d.slice/crio-cdbb42a7c57196e34b8d8ed429fbb178f8a3007df17997c7cbb1f4597bf52fb2 WatchSource:0}: Error finding container cdbb42a7c57196e34b8d8ed429fbb178f8a3007df17997c7cbb1f4597bf52fb2: Status 404 returned error can't find the container with id cdbb42a7c57196e34b8d8ed429fbb178f8a3007df17997c7cbb1f4597bf52fb2
Oct 30 22:08:40.011513 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:40.011145    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56d5748b65-v9kvx"]
Oct 30 22:08:40.015009 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:40.014983    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5ff86975d4-6xkd8"]
Oct 30 22:08:40.017108 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:40.015881    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bdbcb7997-ggr5g"]
Oct 30 22:08:40.020607 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:40.020578    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4369922_ceac_4aaf_9a8b_35c5a0f27d91.slice/crio-744b6d0e11fe91b55509bf9b6d9a575390da464959342e5df95c421ac8d6720d WatchSource:0}: Error finding container 744b6d0e11fe91b55509bf9b6d9a575390da464959342e5df95c421ac8d6720d: Status 404 returned error can't find the container with id 744b6d0e11fe91b55509bf9b6d9a575390da464959342e5df95c421ac8d6720d
Oct 30 22:08:40.061308 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:40.061223    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jg2dx"]
Oct 30 22:08:40.287124 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:40.287095    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t" event={"ID":"ba52011d-6892-4b4d-ac14-07e8be859d9e","Type":"ContainerStarted","Data":"ab004c5e707c7bea77af66a1fa13104d752171dfc8c92dbca950cf997c7f1af8"}
Oct 30 22:08:40.288530 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:40.288485    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jg2dx" event={"ID":"a4369922-ceac-4aaf-9a8b-35c5a0f27d91","Type":"ContainerStarted","Data":"744b6d0e11fe91b55509bf9b6d9a575390da464959342e5df95c421ac8d6720d"}
Oct 30 22:08:40.290268 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:40.290238    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g" event={"ID":"8969718f-1dae-4bde-81df-592ee58a1f76","Type":"ContainerStarted","Data":"fecfb80873534f272366416cc544d9d64d2ae24fe3f2e87499e426febe591579"}
Oct 30 22:08:40.290379 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:40.290293    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g" event={"ID":"8969718f-1dae-4bde-81df-592ee58a1f76","Type":"ContainerStarted","Data":"d2ef17f005a8aebde665278f4ffc8f7da12a76d9fa7a9fa8c87edc0f7909b56b"}
Oct 30 22:08:40.290420 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:40.290386    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:08:40.292033 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:40.291992    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8" event={"ID":"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d","Type":"ContainerStarted","Data":"c314300230f7bc351f0778bfb555751cefee3c2ce443d3739a2e097fdccb37fd"}
Oct 30 22:08:40.292103 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:40.292044    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:08:40.292103 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:40.292067    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8" event={"ID":"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d","Type":"ContainerStarted","Data":"cdbb42a7c57196e34b8d8ed429fbb178f8a3007df17997c7cbb1f4597bf52fb2"}
Oct 30 22:08:40.293667 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:40.293644    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56d5748b65-v9kvx" event={"ID":"94490976-1f2d-4db5-980a-322cad7c057b","Type":"ContainerStarted","Data":"baf21146a732eea98dae7926ed39a349e4727c88a4b4727bbe4a22c9e7ea0e45"}
Oct 30 22:08:40.293730 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:40.293692    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56d5748b65-v9kvx" event={"ID":"94490976-1f2d-4db5-980a-322cad7c057b","Type":"ContainerStarted","Data":"24fdcc6041a820a73cf95097fa5cf1ca1b6296c5b18c0ae639ca5c7b30acd9de"}
Oct 30 22:08:40.458923 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:40.458895    4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:40.461753 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:40.461717    4947 patch_prober.go:29] interesting pod/router-default-56d5748b65-v9kvx container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld
Oct 30 22:08:40.461753 ip-10-0-133-218 kubenswrapper[4947]: [-]has-synced failed: reason withheld
Oct 30 22:08:40.461753 ip-10-0-133-218 kubenswrapper[4947]: [+]process-running ok
Oct 30 22:08:40.461753 ip-10-0-133-218 kubenswrapper[4947]: healthz check failed
Oct 30 22:08:40.462016 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:40.461774    4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-56d5748b65-v9kvx" podUID="94490976-1f2d-4db5-980a-322cad7c057b" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Oct 30 22:08:41.460068 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:41.460043    4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:42.303927 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:42.303889    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-8sk5t" event={"ID":"ba52011d-6892-4b4d-ac14-07e8be859d9e","Type":"ContainerStarted","Data":"8513c7ee7c92055b3a786beb641437db15ea733fdaf818d3ef6bb043cd02fa23"}
Oct 30 22:08:42.305804 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:42.305779    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jg2dx" event={"ID":"a4369922-ceac-4aaf-9a8b-35c5a0f27d91","Type":"ContainerStarted","Data":"c3f8ca53647de177bd5e707d7c27c2e5bcde9d3663a99566bac9dfdae28ed010"}
Oct 30 22:08:42.305957 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:42.305827    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jg2dx" event={"ID":"a4369922-ceac-4aaf-9a8b-35c5a0f27d91","Type":"ContainerStarted","Data":"0bc0ef15b1dd2dd6c7b401d4af79b63683333d66562059420dc3be99c5348916"}
Oct 30 22:08:42.305957 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:42.305871    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:42.305957 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:42.305901    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:42.308668 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:42.308645    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-56d5748b65-v9kvx"
Oct 30 22:08:43.425465 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:43.425424    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bdbcb7997-ggr5g"]
Oct 30 22:08:52.311423 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:52.311396    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jg2dx"
Oct 30 22:08:53.106089 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.106059    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zcmwq"]
Oct 30 22:08:53.150706 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.150679    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.293165 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.293130    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/040c9e13-1604-4a8b-9507-cd2617a1e3e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.293165 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.293166    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/040c9e13-1604-4a8b-9507-cd2617a1e3e2-node-exporter-wtmp\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.293411 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.293195    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/040c9e13-1604-4a8b-9507-cd2617a1e3e2-node-exporter-textfile\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.293411 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.293260    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-957mg\" (UniqueName: \"kubernetes.io/projected/040c9e13-1604-4a8b-9507-cd2617a1e3e2-kube-api-access-957mg\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.293411 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.293377    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/040c9e13-1604-4a8b-9507-cd2617a1e3e2-node-exporter-tls\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.293411 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.293401    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/040c9e13-1604-4a8b-9507-cd2617a1e3e2-metrics-client-ca\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.293595 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.293434    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/040c9e13-1604-4a8b-9507-cd2617a1e3e2-sys\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.293595 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.293475    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/040c9e13-1604-4a8b-9507-cd2617a1e3e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.293595 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.293542    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/040c9e13-1604-4a8b-9507-cd2617a1e3e2-root\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.318023 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.317998    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 22:08:53.318023 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.318002    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 22:08:53.318432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.318023    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 22:08:53.318558 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.318538    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sjbnz\""
Oct 30 22:08:53.318661 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.318640    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 22:08:53.394095 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.394018    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/040c9e13-1604-4a8b-9507-cd2617a1e3e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.394095 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.394056    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/040c9e13-1604-4a8b-9507-cd2617a1e3e2-node-exporter-wtmp\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.394296 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.394096    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/040c9e13-1604-4a8b-9507-cd2617a1e3e2-node-exporter-textfile\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.394296 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.394118    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-957mg\" (UniqueName: \"kubernetes.io/projected/040c9e13-1604-4a8b-9507-cd2617a1e3e2-kube-api-access-957mg\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.394296 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.394154    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/040c9e13-1604-4a8b-9507-cd2617a1e3e2-node-exporter-tls\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.394296 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.394177    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/040c9e13-1604-4a8b-9507-cd2617a1e3e2-metrics-client-ca\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.394296 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.394222    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/040c9e13-1604-4a8b-9507-cd2617a1e3e2-sys\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.394296 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.394247    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/040c9e13-1604-4a8b-9507-cd2617a1e3e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.394296 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.394275    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/040c9e13-1604-4a8b-9507-cd2617a1e3e2-root\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.394296 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.394278    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/040c9e13-1604-4a8b-9507-cd2617a1e3e2-node-exporter-wtmp\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.394687 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.394385    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/040c9e13-1604-4a8b-9507-cd2617a1e3e2-sys\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.394687 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.394397    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/040c9e13-1604-4a8b-9507-cd2617a1e3e2-root\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.394687 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.394554    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/040c9e13-1604-4a8b-9507-cd2617a1e3e2-node-exporter-textfile\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.394975 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.394941    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/040c9e13-1604-4a8b-9507-cd2617a1e3e2-metrics-client-ca\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.394975 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.394941    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/040c9e13-1604-4a8b-9507-cd2617a1e3e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.396641 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.396620    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/040c9e13-1604-4a8b-9507-cd2617a1e3e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.396780 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.396763    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/040c9e13-1604-4a8b-9507-cd2617a1e3e2-node-exporter-tls\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.468630 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.468597    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-957mg\" (UniqueName: \"kubernetes.io/projected/040c9e13-1604-4a8b-9507-cd2617a1e3e2-kube-api-access-957mg\") pod \"node-exporter-zcmwq\" (UID: \"040c9e13-1604-4a8b-9507-cd2617a1e3e2\") " pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.496326 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:53.496290    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zcmwq"
Oct 30 22:08:53.537290 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:08:53.537241    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040c9e13_1604_4a8b_9507_cd2617a1e3e2.slice/crio-81a2d9501c3272712837ea53f1c172e993112d9cf8e2113ba597df95049ad388 WatchSource:0}: Error finding container 81a2d9501c3272712837ea53f1c172e993112d9cf8e2113ba597df95049ad388: Status 404 returned error can't find the container with id 81a2d9501c3272712837ea53f1c172e993112d9cf8e2113ba597df95049ad388
Oct 30 22:08:54.347775 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:54.347744    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zcmwq" event={"ID":"040c9e13-1604-4a8b-9507-cd2617a1e3e2","Type":"ContainerStarted","Data":"81a2d9501c3272712837ea53f1c172e993112d9cf8e2113ba597df95049ad388"}
Oct 30 22:08:54.828909 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:54.828877    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7h6kb"
Oct 30 22:08:55.353666 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:55.353637    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zcmwq" event={"ID":"040c9e13-1604-4a8b-9507-cd2617a1e3e2","Type":"ContainerStarted","Data":"1cb673cdf6d7b1398958e1e5689603da2dcfa4f83becdf21e60fe51afdb63de5"}
Oct 30 22:08:56.358703 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:56.358667    4947 generic.go:356] "Generic (PLEG): container finished" podID="040c9e13-1604-4a8b-9507-cd2617a1e3e2" containerID="1cb673cdf6d7b1398958e1e5689603da2dcfa4f83becdf21e60fe51afdb63de5" exitCode=0
Oct 30 22:08:56.359078 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:56.358733    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zcmwq" event={"ID":"040c9e13-1604-4a8b-9507-cd2617a1e3e2","Type":"ContainerDied","Data":"1cb673cdf6d7b1398958e1e5689603da2dcfa4f83becdf21e60fe51afdb63de5"}
Oct 30 22:08:57.364074 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:57.364038    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zcmwq" event={"ID":"040c9e13-1604-4a8b-9507-cd2617a1e3e2","Type":"ContainerStarted","Data":"45c2d51a0eda363e0220daf62a9ba7ebef078c31db16beed02c09594f3422121"}
Oct 30 22:08:57.364526 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:57.364117    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zcmwq" event={"ID":"040c9e13-1604-4a8b-9507-cd2617a1e3e2","Type":"ContainerStarted","Data":"18a8be370353918ca471bebd177bf2ee08d6b4d5aae811f660f400b2c8fc5086"}
Oct 30 22:08:57.519695 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:08:57.519640    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zcmwq" podStartSLOduration=2.675131607 podStartE2EDuration="4.519627588s" totalImagesPullingTime="1.749279854s" totalInitContainerRuntime="95.216127ms" isStatefulPod=true podCreationTimestamp="2026-10-30 22:08:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:57.48538508 +0000 UTC m=+35.484732643" watchObservedRunningTime="2026-10-30 22:08:57.519627588 +0000 UTC m=+35.518975168"
Oct 30 22:08:59.048096 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:08:59.048064    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:09:01.303499 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:01.303473    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:09:03.152956 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:03.152881    4947 scope.go:122] "RemoveContainer" containerID="3d92772ec7039ec63e1a12df8bcd2e87eabf32e5b9cd00298233567ca6a6f4cf"
Oct 30 22:09:03.386230 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:03.386203    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:09:03.386384 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:03.386306    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr" event={"ID":"00c23fb6-cbfb-4498-ad17-15998e6d2d66","Type":"ContainerStarted","Data":"2f8a7f3f5c1b148bfcf36a271b4dcbdd7e5dfb09d0476f7072f9c1379c6c8ee9"}
Oct 30 22:09:03.386631 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:03.386599    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:09:03.387876 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:03.387857    4947 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-gs4xr container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body=
Oct 30 22:09:03.387971 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:03.387908    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr" podUID="00c23fb6-cbfb-4498-ad17-15998e6d2d66" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused"
Oct 30 22:09:03.432080 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:03.432027    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:09:04.395258 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:04.395225    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gs4xr"
Oct 30 22:09:06.840090 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:06.840061    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5ff86975d4-6xkd8"]
Oct 30 22:09:08.446178 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.446125    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g" podUID="8969718f-1dae-4bde-81df-592ee58a1f76" containerName="registry" containerID="cri-o://fecfb80873534f272366416cc544d9d64d2ae24fe3f2e87499e426febe591579" gracePeriod=30
Oct 30 22:09:08.690691 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.690668    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:09:08.817502 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.817471    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls\") pod \"8969718f-1dae-4bde-81df-592ee58a1f76\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") "
Oct 30 22:09:08.817676 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.817510    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-bound-sa-token\") pod \"8969718f-1dae-4bde-81df-592ee58a1f76\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") "
Oct 30 22:09:08.817676 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.817552    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-registry-certificates\") pod \"8969718f-1dae-4bde-81df-592ee58a1f76\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") "
Oct 30 22:09:08.817676 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.817582    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-kube-api-access-rwmpt\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-kube-api-access-rwmpt\") pod \"8969718f-1dae-4bde-81df-592ee58a1f76\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") "
Oct 30 22:09:08.817676 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.817609    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-trusted-ca\") pod \"8969718f-1dae-4bde-81df-592ee58a1f76\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") "
Oct 30 22:09:08.817676 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.817628    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8969718f-1dae-4bde-81df-592ee58a1f76-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8969718f-1dae-4bde-81df-592ee58a1f76-ca-trust-extracted\") pod \"8969718f-1dae-4bde-81df-592ee58a1f76\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") "
Oct 30 22:09:08.817676 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.817658    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-image-registry-private-configuration\") pod \"8969718f-1dae-4bde-81df-592ee58a1f76\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") "
Oct 30 22:09:08.817998 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.817699    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-installation-pull-secrets\") pod \"8969718f-1dae-4bde-81df-592ee58a1f76\" (UID: \"8969718f-1dae-4bde-81df-592ee58a1f76\") "
Oct 30 22:09:08.818174 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.818102    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-trusted-ca" pod "8969718f-1dae-4bde-81df-592ee58a1f76" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:08.818174 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.818127    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-registry-certificates" pod "8969718f-1dae-4bde-81df-592ee58a1f76" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:08.820449 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.820406    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-bound-sa-token" pod "8969718f-1dae-4bde-81df-592ee58a1f76" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:08.820562 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.820461    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-installation-pull-secrets" pod "8969718f-1dae-4bde-81df-592ee58a1f76" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:08.820562 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.820547    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls" pod "8969718f-1dae-4bde-81df-592ee58a1f76" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:08.820646 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.820568    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-image-registry-private-configuration" pod "8969718f-1dae-4bde-81df-592ee58a1f76" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:08.820808 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.820783    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-kube-api-access-rwmpt" pod "8969718f-1dae-4bde-81df-592ee58a1f76" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76"). InnerVolumeSpecName "kube-api-access-rwmpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:08.825122 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.825099    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8969718f-1dae-4bde-81df-592ee58a1f76-ca-trust-extracted" pod "8969718f-1dae-4bde-81df-592ee58a1f76" (UID: "8969718f-1dae-4bde-81df-592ee58a1f76"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:08.918903 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.918876    4947 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-registry-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:08.918903 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.918902    4947 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-bound-sa-token\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:08.919065 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.918913    4947 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-registry-certificates\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:08.919065 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.918923    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwmpt\" (UniqueName: \"kubernetes.io/projected/8969718f-1dae-4bde-81df-592ee58a1f76-kube-api-access-rwmpt\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:08.919065 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.918937    4947 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8969718f-1dae-4bde-81df-592ee58a1f76-trusted-ca\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:08.919065 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.918949    4947 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8969718f-1dae-4bde-81df-592ee58a1f76-ca-trust-extracted\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:08.919065 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.918959    4947 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-image-registry-private-configuration\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:08.919065 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:08.918968    4947 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8969718f-1dae-4bde-81df-592ee58a1f76-installation-pull-secrets\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:09.410523 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:09.410492    4947 generic.go:356] "Generic (PLEG): container finished" podID="8969718f-1dae-4bde-81df-592ee58a1f76" containerID="fecfb80873534f272366416cc544d9d64d2ae24fe3f2e87499e426febe591579" exitCode=0
Oct 30 22:09:09.410682 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:09.410528    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g" event={"ID":"8969718f-1dae-4bde-81df-592ee58a1f76","Type":"ContainerDied","Data":"fecfb80873534f272366416cc544d9d64d2ae24fe3f2e87499e426febe591579"}
Oct 30 22:09:09.410682 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:09.410540    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g"
Oct 30 22:09:09.410682 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:09.410568    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bdbcb7997-ggr5g" event={"ID":"8969718f-1dae-4bde-81df-592ee58a1f76","Type":"ContainerDied","Data":"d2ef17f005a8aebde665278f4ffc8f7da12a76d9fa7a9fa8c87edc0f7909b56b"}
Oct 30 22:09:09.410682 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:09.410586    4947 scope.go:122] "RemoveContainer" containerID="fecfb80873534f272366416cc544d9d64d2ae24fe3f2e87499e426febe591579"
Oct 30 22:09:09.419567 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:09.419545    4947 scope.go:122] "RemoveContainer" containerID="fecfb80873534f272366416cc544d9d64d2ae24fe3f2e87499e426febe591579"
Oct 30 22:09:09.419879 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:09:09.419859    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fecfb80873534f272366416cc544d9d64d2ae24fe3f2e87499e426febe591579\": container with ID starting with fecfb80873534f272366416cc544d9d64d2ae24fe3f2e87499e426febe591579 not found: ID does not exist" containerID="fecfb80873534f272366416cc544d9d64d2ae24fe3f2e87499e426febe591579"
Oct 30 22:09:09.419952 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:09.419887    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fecfb80873534f272366416cc544d9d64d2ae24fe3f2e87499e426febe591579"} err="failed to get container status \"fecfb80873534f272366416cc544d9d64d2ae24fe3f2e87499e426febe591579\": rpc error: code = NotFound desc = could not find container \"fecfb80873534f272366416cc544d9d64d2ae24fe3f2e87499e426febe591579\": container with ID starting with fecfb80873534f272366416cc544d9d64d2ae24fe3f2e87499e426febe591579 not found: ID does not exist"
Oct 30 22:09:09.627386 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:09.627348    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bdbcb7997-ggr5g"]
Oct 30 22:09:09.676791 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:09.676723    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-bdbcb7997-ggr5g"]
Oct 30 22:09:10.157196 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:10.157159    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8969718f-1dae-4bde-81df-592ee58a1f76" path="/var/lib/kubelet/pods/8969718f-1dae-4bde-81df-592ee58a1f76/volumes"
Oct 30 22:09:29.048565 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:09:29.048527    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:09:31.861313 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:31.861278    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8" podUID="4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" containerName="registry" containerID="cri-o://c314300230f7bc351f0778bfb555751cefee3c2ce443d3739a2e097fdccb37fd" gracePeriod=30
Oct 30 22:09:32.110945 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.110921    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:09:32.185591 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.185534    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-installation-pull-secrets\") pod \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") "
Oct 30 22:09:32.185591 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.185567    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-trusted-ca\") pod \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") "
Oct 30 22:09:32.185788 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.185597    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-ca-trust-extracted\") pod \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") "
Oct 30 22:09:32.185788 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.185638    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-kube-api-access-fx7tp\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-kube-api-access-fx7tp\") pod \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") "
Oct 30 22:09:32.185788 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.185674    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-certificates\") pod \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") "
Oct 30 22:09:32.185788 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.185712    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-image-registry-private-configuration\") pod \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") "
Oct 30 22:09:32.185788 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.185748    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls\") pod \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") "
Oct 30 22:09:32.185788 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.185777    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-bound-sa-token\") pod \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\" (UID: \"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d\") "
Oct 30 22:09:32.186101 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.186024    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-trusted-ca" pod "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:32.186711 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.186636    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-certificates" pod "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:32.188105 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.188082    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-installation-pull-secrets" pod "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:32.188296 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.188270    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-bound-sa-token" pod "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:32.188677 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.188642    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-kube-api-access-fx7tp" pod "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d"). InnerVolumeSpecName "kube-api-access-fx7tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:32.189466 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.189441    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls" pod "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:32.189543 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.189509    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-image-registry-private-configuration" pod "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:32.192991 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.192964    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-ca-trust-extracted" pod "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" (UID: "4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:32.287187 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.287161    4947 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-certificates\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:32.287187 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.287184    4947 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-image-registry-private-configuration\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:32.287353 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.287194    4947 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-registry-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:32.287353 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.287204    4947 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-bound-sa-token\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:32.287353 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.287212    4947 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-installation-pull-secrets\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:32.287353 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.287221    4947 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-trusted-ca\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:32.287353 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.287229    4947 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-ca-trust-extracted\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:32.287353 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.287237    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx7tp\" (UniqueName: \"kubernetes.io/projected/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d-kube-api-access-fx7tp\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:32.494615 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.494539    4947 generic.go:356] "Generic (PLEG): container finished" podID="4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" containerID="c314300230f7bc351f0778bfb555751cefee3c2ce443d3739a2e097fdccb37fd" exitCode=0
Oct 30 22:09:32.494615 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.494588    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8"
Oct 30 22:09:32.494615 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.494595    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8" event={"ID":"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d","Type":"ContainerDied","Data":"c314300230f7bc351f0778bfb555751cefee3c2ce443d3739a2e097fdccb37fd"}
Oct 30 22:09:32.494802 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.494641    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ff86975d4-6xkd8" event={"ID":"4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d","Type":"ContainerDied","Data":"cdbb42a7c57196e34b8d8ed429fbb178f8a3007df17997c7cbb1f4597bf52fb2"}
Oct 30 22:09:32.494802 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.494670    4947 scope.go:122] "RemoveContainer" containerID="c314300230f7bc351f0778bfb555751cefee3c2ce443d3739a2e097fdccb37fd"
Oct 30 22:09:32.503265 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.503249    4947 scope.go:122] "RemoveContainer" containerID="c314300230f7bc351f0778bfb555751cefee3c2ce443d3739a2e097fdccb37fd"
Oct 30 22:09:32.503549 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:09:32.503530    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c314300230f7bc351f0778bfb555751cefee3c2ce443d3739a2e097fdccb37fd\": container with ID starting with c314300230f7bc351f0778bfb555751cefee3c2ce443d3739a2e097fdccb37fd not found: ID does not exist" containerID="c314300230f7bc351f0778bfb555751cefee3c2ce443d3739a2e097fdccb37fd"
Oct 30 22:09:32.503640 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.503557    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c314300230f7bc351f0778bfb555751cefee3c2ce443d3739a2e097fdccb37fd"} err="failed to get container status \"c314300230f7bc351f0778bfb555751cefee3c2ce443d3739a2e097fdccb37fd\": rpc error: code = NotFound desc = could not find container \"c314300230f7bc351f0778bfb555751cefee3c2ce443d3739a2e097fdccb37fd\": container with ID starting with c314300230f7bc351f0778bfb555751cefee3c2ce443d3739a2e097fdccb37fd not found: ID does not exist"
Oct 30 22:09:32.660492 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.660459    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5ff86975d4-6xkd8"]
Oct 30 22:09:32.666313 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:32.666293    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5ff86975d4-6xkd8"]
Oct 30 22:09:34.158136 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:34.158100    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" path="/var/lib/kubelet/pods/4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d/volumes"
Oct 30 22:09:37.516344 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:37.516310    4947 generic.go:356] "Generic (PLEG): container finished" podID="a4254134-79fd-411a-a492-3e620d223ddf" containerID="412740f4f705faabbe68c8ea351e4d2f4f4d9c4a34f7bb30ca614a8a99438210" exitCode=0
Oct 30 22:09:37.516731 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:37.516389    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-652qc" event={"ID":"a4254134-79fd-411a-a492-3e620d223ddf","Type":"ContainerDied","Data":"412740f4f705faabbe68c8ea351e4d2f4f4d9c4a34f7bb30ca614a8a99438210"}
Oct 30 22:09:37.516774 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:37.516753    4947 scope.go:122] "RemoveContainer" containerID="412740f4f705faabbe68c8ea351e4d2f4f4d9c4a34f7bb30ca614a8a99438210"
Oct 30 22:09:38.520942 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:38.520904    4947 generic.go:356] "Generic (PLEG): container finished" podID="09b48430-dd49-439d-b317-384408fec509" containerID="c19c7feede0fd2eb742fa41a05be6e9f0e9b53c3ee5c7eb4e8d97bdba4555e15" exitCode=0
Oct 30 22:09:38.521385 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:38.520974    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp" event={"ID":"09b48430-dd49-439d-b317-384408fec509","Type":"ContainerDied","Data":"c19c7feede0fd2eb742fa41a05be6e9f0e9b53c3ee5c7eb4e8d97bdba4555e15"}
Oct 30 22:09:38.521385 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:38.521306    4947 scope.go:122] "RemoveContainer" containerID="c19c7feede0fd2eb742fa41a05be6e9f0e9b53c3ee5c7eb4e8d97bdba4555e15"
Oct 30 22:09:38.522842 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:38.522783    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-652qc" event={"ID":"a4254134-79fd-411a-a492-3e620d223ddf","Type":"ContainerStarted","Data":"d44f3fde57a842c5dd14c4d3687d0daf5e7c4b1360b3f44058319ab353c596fe"}
Oct 30 22:09:39.528887 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:39.528855    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6vlp" event={"ID":"09b48430-dd49-439d-b317-384408fec509","Type":"ContainerStarted","Data":"d5fdcd2b2c2498bcae0d3a01cdf085bfb4c64e3351d93d992f0a15a762348cf9"}
Oct 30 22:09:49.564343 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:49.564293    4947 generic.go:356] "Generic (PLEG): container finished" podID="bb5baa5b-c887-42ea-b759-9347ae332051" containerID="a46b5a06f654ad29b8481476b2c6367c02e1249bd54bb655b102ad6c110e636c" exitCode=0
Oct 30 22:09:49.564695 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:49.564370    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7" event={"ID":"bb5baa5b-c887-42ea-b759-9347ae332051","Type":"ContainerDied","Data":"a46b5a06f654ad29b8481476b2c6367c02e1249bd54bb655b102ad6c110e636c"}
Oct 30 22:09:49.564735 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:49.564701    4947 scope.go:122] "RemoveContainer" containerID="a46b5a06f654ad29b8481476b2c6367c02e1249bd54bb655b102ad6c110e636c"
Oct 30 22:09:50.569414 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:09:50.569380    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-js4z7" event={"ID":"bb5baa5b-c887-42ea-b759-9347ae332051","Type":"ContainerStarted","Data":"64d402c42e6330e92be31f31d671035316112be88c5a0b051963d254d4c61131"}
Oct 30 22:09:59.048410 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:09:59.048371    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:10:29.047850 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:10:29.047810    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:10:59.047702 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:10:59.047629    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:11:29.047819 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:11:29.047783    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:11:59.047226 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:11:59.047188    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:12:29.048235 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:12:29.048200    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:12:59.048683 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:12:59.048643    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:12:59.901758 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:12:59.901726    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-7mdx8"]
Oct 30 22:12:59.902033 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:12:59.902019    4947 cpu_manager.go:414] "containerMap: removing container" podUID="4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" containerName="registry"
Oct 30 22:12:59.902091 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:12:59.902034    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" containerName="registry"
Oct 30 22:12:59.902091 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:12:59.902059    4947 cpu_manager.go:414] "containerMap: removing container" podUID="8969718f-1dae-4bde-81df-592ee58a1f76" containerName="registry"
Oct 30 22:12:59.902091 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:12:59.902064    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8969718f-1dae-4bde-81df-592ee58a1f76" containerName="registry"
Oct 30 22:12:59.902196 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:12:59.902111    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c0035aa-74bd-417b-ba1e-0ebcb58c6e3d" containerName="registry"
Oct 30 22:12:59.902196 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:12:59.902121    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="8969718f-1dae-4bde-81df-592ee58a1f76" containerName="registry"
Oct 30 22:12:59.904979 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:12:59.904959    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-7b7c985c58-7mdx8"
Oct 30 22:13:00.007622 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:00.007590    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-4h588\""
Oct 30 22:13:00.007622 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:00.007591    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Oct 30 22:13:00.007622 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:00.007621    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Oct 30 22:13:00.007871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:00.007621    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Oct 30 22:13:00.007871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:00.007591    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Oct 30 22:13:00.009226 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:00.009202    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-7mdx8"]
Oct 30 22:13:00.016616 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:00.016590    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/948dd7f5-0ac2-47d2-ac57-cfad45f7045c-certificates\") pod \"keda-admission-7b7c985c58-7mdx8\" (UID: \"948dd7f5-0ac2-47d2-ac57-cfad45f7045c\") " pod="openshift-keda/keda-admission-7b7c985c58-7mdx8"
Oct 30 22:13:00.016690 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:00.016650    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwmf5\" (UniqueName: \"kubernetes.io/projected/948dd7f5-0ac2-47d2-ac57-cfad45f7045c-kube-api-access-jwmf5\") pod \"keda-admission-7b7c985c58-7mdx8\" (UID: \"948dd7f5-0ac2-47d2-ac57-cfad45f7045c\") " pod="openshift-keda/keda-admission-7b7c985c58-7mdx8"
Oct 30 22:13:00.118027 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:00.117985    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwmf5\" (UniqueName: \"kubernetes.io/projected/948dd7f5-0ac2-47d2-ac57-cfad45f7045c-kube-api-access-jwmf5\") pod \"keda-admission-7b7c985c58-7mdx8\" (UID: \"948dd7f5-0ac2-47d2-ac57-cfad45f7045c\") " pod="openshift-keda/keda-admission-7b7c985c58-7mdx8"
Oct 30 22:13:00.118434 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:00.118044    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/948dd7f5-0ac2-47d2-ac57-cfad45f7045c-certificates\") pod \"keda-admission-7b7c985c58-7mdx8\" (UID: \"948dd7f5-0ac2-47d2-ac57-cfad45f7045c\") " pod="openshift-keda/keda-admission-7b7c985c58-7mdx8"
Oct 30 22:13:00.120633 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:00.120614    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/948dd7f5-0ac2-47d2-ac57-cfad45f7045c-certificates\") pod \"keda-admission-7b7c985c58-7mdx8\" (UID: \"948dd7f5-0ac2-47d2-ac57-cfad45f7045c\") " pod="openshift-keda/keda-admission-7b7c985c58-7mdx8"
Oct 30 22:13:00.190141 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:00.190075    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwmf5\" (UniqueName: \"kubernetes.io/projected/948dd7f5-0ac2-47d2-ac57-cfad45f7045c-kube-api-access-jwmf5\") pod \"keda-admission-7b7c985c58-7mdx8\" (UID: \"948dd7f5-0ac2-47d2-ac57-cfad45f7045c\") " pod="openshift-keda/keda-admission-7b7c985c58-7mdx8"
Oct 30 22:13:00.216903 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:00.216877    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-7b7c985c58-7mdx8"
Oct 30 22:13:00.590142 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:00.590112    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-7mdx8"]
Oct 30 22:13:00.593500 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:13:00.593472    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948dd7f5_0ac2_47d2_ac57_cfad45f7045c.slice/crio-933a11f7e1c5b534432306b10af8f4cf9da5e1be4f4c8e9c55393a5f273585c5 WatchSource:0}: Error finding container 933a11f7e1c5b534432306b10af8f4cf9da5e1be4f4c8e9c55393a5f273585c5: Status 404 returned error can't find the container with id 933a11f7e1c5b534432306b10af8f4cf9da5e1be4f4c8e9c55393a5f273585c5
Oct 30 22:13:01.238408 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:01.238372    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-7b7c985c58-7mdx8" event={"ID":"948dd7f5-0ac2-47d2-ac57-cfad45f7045c","Type":"ContainerStarted","Data":"933a11f7e1c5b534432306b10af8f4cf9da5e1be4f4c8e9c55393a5f273585c5"}
Oct 30 22:13:05.256708 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:05.256613    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-7b7c985c58-7mdx8" event={"ID":"948dd7f5-0ac2-47d2-ac57-cfad45f7045c","Type":"ContainerStarted","Data":"a2aba8e9b9767d32e80b434e5054a26c033d102f2f3ceca6ecb74470fe603d35"}
Oct 30 22:13:05.257136 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:05.256765    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-7b7c985c58-7mdx8"
Oct 30 22:13:05.384855 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:05.384800    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-admission-7b7c985c58-7mdx8" podStartSLOduration=2.109975394 podStartE2EDuration="6.384784956s" totalImagesPullingTime="4.274809562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:05.383579873 +0000 UTC m=+283.382927436" watchObservedRunningTime="2026-10-30 22:13:05.384784956 +0000 UTC m=+283.384132519"
Oct 30 22:13:22.157255 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:22.157229    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:13:22.157758 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:22.157236    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:13:26.261869 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:13:26.261838    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-7b7c985c58-7mdx8"
Oct 30 22:13:29.048156 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:13:29.048120    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:13:59.047745 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:13:59.047662    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:14:23.475621 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:23.475589    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-2jk46"]
Oct 30 22:14:23.478876 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:23.478856    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46"
Oct 30 22:14:23.532620 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:23.532597    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Oct 30 22:14:23.581917 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:23.581890    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 30 22:14:23.583042 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:23.583021    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-2jk46"]
Oct 30 22:14:23.634186 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:23.634159    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-r945v\""
Oct 30 22:14:23.637046 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:23.637028    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 30 22:14:23.664365 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:23.664321    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdd89\" (UniqueName: \"kubernetes.io/projected/ca427b29-b5d4-4489-baeb-6fa81d90e20a-kube-api-access-vdd89\") pod \"kserve-controller-manager-76d8b77fbb-2jk46\" (UID: \"ca427b29-b5d4-4489-baeb-6fa81d90e20a\") " pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46"
Oct 30 22:14:23.664456 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:23.664400    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca427b29-b5d4-4489-baeb-6fa81d90e20a-cert\") pod \"kserve-controller-manager-76d8b77fbb-2jk46\" (UID: \"ca427b29-b5d4-4489-baeb-6fa81d90e20a\") " pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46"
Oct 30 22:14:23.765314 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:23.765238    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca427b29-b5d4-4489-baeb-6fa81d90e20a-cert\") pod \"kserve-controller-manager-76d8b77fbb-2jk46\" (UID: \"ca427b29-b5d4-4489-baeb-6fa81d90e20a\") " pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46"
Oct 30 22:14:23.765314 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:23.765289    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdd89\" (UniqueName: \"kubernetes.io/projected/ca427b29-b5d4-4489-baeb-6fa81d90e20a-kube-api-access-vdd89\") pod \"kserve-controller-manager-76d8b77fbb-2jk46\" (UID: \"ca427b29-b5d4-4489-baeb-6fa81d90e20a\") " pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46"
Oct 30 22:14:23.767751 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:23.767733    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca427b29-b5d4-4489-baeb-6fa81d90e20a-cert\") pod \"kserve-controller-manager-76d8b77fbb-2jk46\" (UID: \"ca427b29-b5d4-4489-baeb-6fa81d90e20a\") " pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46"
Oct 30 22:14:23.828177 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:23.828144    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdd89\" (UniqueName: \"kubernetes.io/projected/ca427b29-b5d4-4489-baeb-6fa81d90e20a-kube-api-access-vdd89\") pod \"kserve-controller-manager-76d8b77fbb-2jk46\" (UID: \"ca427b29-b5d4-4489-baeb-6fa81d90e20a\") " pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46"
Oct 30 22:14:24.090749 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:24.090704    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46"
Oct 30 22:14:24.482739 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:24.482663    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-2jk46"]
Oct 30 22:14:24.485919 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:14:24.485893    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca427b29_b5d4_4489_baeb_6fa81d90e20a.slice/crio-b881caad86a6439d750f90f91252125ec7a727462eccb48d25f1f25ed7f1dde5 WatchSource:0}: Error finding container b881caad86a6439d750f90f91252125ec7a727462eccb48d25f1f25ed7f1dde5: Status 404 returned error can't find the container with id b881caad86a6439d750f90f91252125ec7a727462eccb48d25f1f25ed7f1dde5
Oct 30 22:14:24.487624 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:24.487605    4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:14:24.539716 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:24.539689    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46" event={"ID":"ca427b29-b5d4-4489-baeb-6fa81d90e20a","Type":"ContainerStarted","Data":"b881caad86a6439d750f90f91252125ec7a727462eccb48d25f1f25ed7f1dde5"}
Oct 30 22:14:29.048134 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:14:29.048094    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:14:33.575322 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:33.575286    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46" event={"ID":"ca427b29-b5d4-4489-baeb-6fa81d90e20a","Type":"ContainerStarted","Data":"2defb0bbbf87f954a100273c363f9dcb1e7cda0e996a9e4e81b393bbf87ccd91"}
Oct 30 22:14:33.575767 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:33.575397    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46"
Oct 30 22:14:33.695975 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:14:33.695921    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46" podStartSLOduration=2.3851398440000002 podStartE2EDuration="10.695907256s" totalImagesPullingTime="8.310767412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:33.694990928 +0000 UTC m=+371.694338491" watchObservedRunningTime="2026-10-30 22:14:33.695907256 +0000 UTC m=+371.695254796"
Oct 30 22:14:59.048765 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:14:59.048727    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:15:04.583034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:04.582965    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46"
Oct 30 22:15:21.622883 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.622845    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-2jk46"]
Oct 30 22:15:21.623360 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.623050    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46" podUID="ca427b29-b5d4-4489-baeb-6fa81d90e20a" containerName="manager" containerID="cri-o://2defb0bbbf87f954a100273c363f9dcb1e7cda0e996a9e4e81b393bbf87ccd91" gracePeriod=10
Oct 30 22:15:21.660870 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.660840    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-8c8jr"]
Oct 30 22:15:21.664464 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.664449    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-8c8jr"
Oct 30 22:15:21.736952 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.736927    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-8c8jr"]
Oct 30 22:15:21.744565 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.744540    4947 generic.go:356] "Generic (PLEG): container finished" podID="ca427b29-b5d4-4489-baeb-6fa81d90e20a" containerID="2defb0bbbf87f954a100273c363f9dcb1e7cda0e996a9e4e81b393bbf87ccd91" exitCode=0
Oct 30 22:15:21.744684 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.744598    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46" event={"ID":"ca427b29-b5d4-4489-baeb-6fa81d90e20a","Type":"ContainerDied","Data":"2defb0bbbf87f954a100273c363f9dcb1e7cda0e996a9e4e81b393bbf87ccd91"}
Oct 30 22:15:21.793857 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.793823    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64b86a66-bd12-4a8f-9d81-53ad0e59207a-cert\") pod \"kserve-controller-manager-76d8b77fbb-8c8jr\" (UID: \"64b86a66-bd12-4a8f-9d81-53ad0e59207a\") " pod="kserve/kserve-controller-manager-76d8b77fbb-8c8jr"
Oct 30 22:15:21.793997 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.793892    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7w7k\" (UniqueName: \"kubernetes.io/projected/64b86a66-bd12-4a8f-9d81-53ad0e59207a-kube-api-access-n7w7k\") pod \"kserve-controller-manager-76d8b77fbb-8c8jr\" (UID: \"64b86a66-bd12-4a8f-9d81-53ad0e59207a\") " pod="kserve/kserve-controller-manager-76d8b77fbb-8c8jr"
Oct 30 22:15:21.868111 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.868085    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46"
Oct 30 22:15:21.894704 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.894645    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64b86a66-bd12-4a8f-9d81-53ad0e59207a-cert\") pod \"kserve-controller-manager-76d8b77fbb-8c8jr\" (UID: \"64b86a66-bd12-4a8f-9d81-53ad0e59207a\") " pod="kserve/kserve-controller-manager-76d8b77fbb-8c8jr"
Oct 30 22:15:21.894704 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.894689    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7w7k\" (UniqueName: \"kubernetes.io/projected/64b86a66-bd12-4a8f-9d81-53ad0e59207a-kube-api-access-n7w7k\") pod \"kserve-controller-manager-76d8b77fbb-8c8jr\" (UID: \"64b86a66-bd12-4a8f-9d81-53ad0e59207a\") " pod="kserve/kserve-controller-manager-76d8b77fbb-8c8jr"
Oct 30 22:15:21.897222 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.897184    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64b86a66-bd12-4a8f-9d81-53ad0e59207a-cert\") pod \"kserve-controller-manager-76d8b77fbb-8c8jr\" (UID: \"64b86a66-bd12-4a8f-9d81-53ad0e59207a\") " pod="kserve/kserve-controller-manager-76d8b77fbb-8c8jr"
Oct 30 22:15:21.965535 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.965505    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7w7k\" (UniqueName: \"kubernetes.io/projected/64b86a66-bd12-4a8f-9d81-53ad0e59207a-kube-api-access-n7w7k\") pod \"kserve-controller-manager-76d8b77fbb-8c8jr\" (UID: \"64b86a66-bd12-4a8f-9d81-53ad0e59207a\") " pod="kserve/kserve-controller-manager-76d8b77fbb-8c8jr"
Oct 30 22:15:21.995790 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.995761    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca427b29-b5d4-4489-baeb-6fa81d90e20a-kube-api-access-vdd89\" (UniqueName: \"kubernetes.io/projected/ca427b29-b5d4-4489-baeb-6fa81d90e20a-kube-api-access-vdd89\") pod \"ca427b29-b5d4-4489-baeb-6fa81d90e20a\" (UID: \"ca427b29-b5d4-4489-baeb-6fa81d90e20a\") "
Oct 30 22:15:21.995930 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.995816    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca427b29-b5d4-4489-baeb-6fa81d90e20a-cert\" (UniqueName: \"kubernetes.io/secret/ca427b29-b5d4-4489-baeb-6fa81d90e20a-cert\") pod \"ca427b29-b5d4-4489-baeb-6fa81d90e20a\" (UID: \"ca427b29-b5d4-4489-baeb-6fa81d90e20a\") "
Oct 30 22:15:21.998355 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.998303    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca427b29-b5d4-4489-baeb-6fa81d90e20a-kube-api-access-vdd89" pod "ca427b29-b5d4-4489-baeb-6fa81d90e20a" (UID: "ca427b29-b5d4-4489-baeb-6fa81d90e20a"). InnerVolumeSpecName "kube-api-access-vdd89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:21.998355 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:21.998352    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca427b29-b5d4-4489-baeb-6fa81d90e20a-cert" pod "ca427b29-b5d4-4489-baeb-6fa81d90e20a" (UID: "ca427b29-b5d4-4489-baeb-6fa81d90e20a"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:22.001427 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:22.001402    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-8c8jr"
Oct 30 22:15:22.096542 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:22.096514    4947 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca427b29-b5d4-4489-baeb-6fa81d90e20a-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:22.096542 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:22.096540    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdd89\" (UniqueName: \"kubernetes.io/projected/ca427b29-b5d4-4489-baeb-6fa81d90e20a-kube-api-access-vdd89\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:22.383235 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:22.383048    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-8c8jr"]
Oct 30 22:15:22.385805 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:15:22.385780    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b86a66_bd12_4a8f_9d81_53ad0e59207a.slice/crio-43aa3e9deccda1fc219926512d7d630f03ec5ffad9a6f22fb7f85bf2e2680903 WatchSource:0}: Error finding container 43aa3e9deccda1fc219926512d7d630f03ec5ffad9a6f22fb7f85bf2e2680903: Status 404 returned error can't find the container with id 43aa3e9deccda1fc219926512d7d630f03ec5ffad9a6f22fb7f85bf2e2680903
Oct 30 22:15:22.749347 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:22.749301    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46"
Oct 30 22:15:22.749822 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:22.749351    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-2jk46" event={"ID":"ca427b29-b5d4-4489-baeb-6fa81d90e20a","Type":"ContainerDied","Data":"b881caad86a6439d750f90f91252125ec7a727462eccb48d25f1f25ed7f1dde5"}
Oct 30 22:15:22.749822 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:22.749426    4947 scope.go:122] "RemoveContainer" containerID="2defb0bbbf87f954a100273c363f9dcb1e7cda0e996a9e4e81b393bbf87ccd91"
Oct 30 22:15:22.750712 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:22.750647    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-8c8jr" event={"ID":"64b86a66-bd12-4a8f-9d81-53ad0e59207a","Type":"ContainerStarted","Data":"43aa3e9deccda1fc219926512d7d630f03ec5ffad9a6f22fb7f85bf2e2680903"}
Oct 30 22:15:22.915469 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:22.915388    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-2jk46"]
Oct 30 22:15:22.918150 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:22.918126    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-2jk46"]
Oct 30 22:15:23.756731 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:23.756643    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-8c8jr" event={"ID":"64b86a66-bd12-4a8f-9d81-53ad0e59207a","Type":"ContainerStarted","Data":"a29ca877f557b5573e626ea03b9f75aac286b035e789e67fc7c94cf245f10292"}
Oct 30 22:15:23.757096 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:23.756779    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-76d8b77fbb-8c8jr"
Oct 30 22:15:23.876951 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:23.876792    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/kserve-controller-manager-76d8b77fbb-8c8jr" podStartSLOduration=1.9484169900000001 podStartE2EDuration="2.876775049s" totalImagesPullingTime="928.358059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:23.876306048 +0000 UTC m=+421.875653611" watchObservedRunningTime="2026-10-30 22:15:23.876775049 +0000 UTC m=+421.876122610"
Oct 30 22:15:24.158064 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:24.158030    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca427b29-b5d4-4489-baeb-6fa81d90e20a" path="/var/lib/kubelet/pods/ca427b29-b5d4-4489-baeb-6fa81d90e20a/volumes"
Oct 30 22:15:29.047753 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:15:29.047715    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:15:54.766055 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:54.766026    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-76d8b77fbb-8c8jr"
Oct 30 22:15:56.900012 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:56.899981    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-56ddc4c448-f6qxz"]
Oct 30 22:15:56.900408 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:56.900304    4947 cpu_manager.go:414] "containerMap: removing container" podUID="ca427b29-b5d4-4489-baeb-6fa81d90e20a" containerName="manager"
Oct 30 22:15:56.900408 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:56.900315    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca427b29-b5d4-4489-baeb-6fa81d90e20a" containerName="manager"
Oct 30 22:15:56.900408 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:56.900380    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca427b29-b5d4-4489-baeb-6fa81d90e20a" containerName="manager"
Oct 30 22:15:56.903257 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:56.903241    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-f6qxz"
Oct 30 22:15:56.947926 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:56.947899    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-6m8w7"]
Oct 30 22:15:56.950953 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:56.950937    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-6m8w7"
Oct 30 22:15:56.996461 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:56.996434    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Oct 30 22:15:56.996593 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:56.996478    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-jhd8x\""
Oct 30 22:15:56.997650 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:56.997630    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-f6qxz"]
Oct 30 22:15:57.010847 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.010824    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-6m8w7"]
Oct 30 22:15:57.012950 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.012930    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Oct 30 22:15:57.013055 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.013009    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-vlkpv\""
Oct 30 22:15:57.046366 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.046323    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Oct 30 22:15:57.069795 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.069775    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be12cdf3-ae3b-469b-a303-dd178829d1e5-cert\") pod \"odh-model-controller-7f9f8cdc65-6m8w7\" (UID: \"be12cdf3-ae3b-469b-a303-dd178829d1e5\") " pod="kserve/odh-model-controller-7f9f8cdc65-6m8w7"
Oct 30 22:15:57.069909 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.069809    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk7zm\" (UniqueName: \"kubernetes.io/projected/be12cdf3-ae3b-469b-a303-dd178829d1e5-kube-api-access-zk7zm\") pod \"odh-model-controller-7f9f8cdc65-6m8w7\" (UID: \"be12cdf3-ae3b-469b-a303-dd178829d1e5\") " pod="kserve/odh-model-controller-7f9f8cdc65-6m8w7"
Oct 30 22:15:57.069909 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.069830    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be12cdf3-ae3b-469b-a303-dd178829d1e5-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-6m8w7\" (UID: \"be12cdf3-ae3b-469b-a303-dd178829d1e5\") " pod="kserve/odh-model-controller-7f9f8cdc65-6m8w7"
Oct 30 22:15:57.069909 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.069853    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7adcbc6-56d2-4e03-88b0-57ef2cae32d4-tls-certs\") pod \"model-serving-api-56ddc4c448-f6qxz\" (UID: \"c7adcbc6-56d2-4e03-88b0-57ef2cae32d4\") " pod="kserve/model-serving-api-56ddc4c448-f6qxz"
Oct 30 22:15:57.070042 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.070007    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vql85\" (UniqueName: \"kubernetes.io/projected/c7adcbc6-56d2-4e03-88b0-57ef2cae32d4-kube-api-access-vql85\") pod \"model-serving-api-56ddc4c448-f6qxz\" (UID: \"c7adcbc6-56d2-4e03-88b0-57ef2cae32d4\") " pod="kserve/model-serving-api-56ddc4c448-f6qxz"
Oct 30 22:15:57.170451 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.170391    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be12cdf3-ae3b-469b-a303-dd178829d1e5-cert\") pod \"odh-model-controller-7f9f8cdc65-6m8w7\" (UID: \"be12cdf3-ae3b-469b-a303-dd178829d1e5\") " pod="kserve/odh-model-controller-7f9f8cdc65-6m8w7"
Oct 30 22:15:57.170451 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.170428    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk7zm\" (UniqueName: \"kubernetes.io/projected/be12cdf3-ae3b-469b-a303-dd178829d1e5-kube-api-access-zk7zm\") pod \"odh-model-controller-7f9f8cdc65-6m8w7\" (UID: \"be12cdf3-ae3b-469b-a303-dd178829d1e5\") " pod="kserve/odh-model-controller-7f9f8cdc65-6m8w7"
Oct 30 22:15:57.170634 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.170450    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be12cdf3-ae3b-469b-a303-dd178829d1e5-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-6m8w7\" (UID: \"be12cdf3-ae3b-469b-a303-dd178829d1e5\") " pod="kserve/odh-model-controller-7f9f8cdc65-6m8w7"
Oct 30 22:15:57.170634 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.170473    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7adcbc6-56d2-4e03-88b0-57ef2cae32d4-tls-certs\") pod \"model-serving-api-56ddc4c448-f6qxz\" (UID: \"c7adcbc6-56d2-4e03-88b0-57ef2cae32d4\") " pod="kserve/model-serving-api-56ddc4c448-f6qxz"
Oct 30 22:15:57.170634 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.170514    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vql85\" (UniqueName: \"kubernetes.io/projected/c7adcbc6-56d2-4e03-88b0-57ef2cae32d4-kube-api-access-vql85\") pod \"model-serving-api-56ddc4c448-f6qxz\" (UID: \"c7adcbc6-56d2-4e03-88b0-57ef2cae32d4\") " pod="kserve/model-serving-api-56ddc4c448-f6qxz"
Oct 30 22:15:57.172977 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.172944    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be12cdf3-ae3b-469b-a303-dd178829d1e5-cert\") pod \"odh-model-controller-7f9f8cdc65-6m8w7\" (UID: \"be12cdf3-ae3b-469b-a303-dd178829d1e5\") " pod="kserve/odh-model-controller-7f9f8cdc65-6m8w7"
Oct 30 22:15:57.173082 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.173031    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7adcbc6-56d2-4e03-88b0-57ef2cae32d4-tls-certs\") pod \"model-serving-api-56ddc4c448-f6qxz\" (UID: \"c7adcbc6-56d2-4e03-88b0-57ef2cae32d4\") " pod="kserve/model-serving-api-56ddc4c448-f6qxz"
Oct 30 22:15:57.173128 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.173078    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be12cdf3-ae3b-469b-a303-dd178829d1e5-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-6m8w7\" (UID: \"be12cdf3-ae3b-469b-a303-dd178829d1e5\") " pod="kserve/odh-model-controller-7f9f8cdc65-6m8w7"
Oct 30 22:15:57.238160 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.238120    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk7zm\" (UniqueName: \"kubernetes.io/projected/be12cdf3-ae3b-469b-a303-dd178829d1e5-kube-api-access-zk7zm\") pod \"odh-model-controller-7f9f8cdc65-6m8w7\" (UID: \"be12cdf3-ae3b-469b-a303-dd178829d1e5\") " pod="kserve/odh-model-controller-7f9f8cdc65-6m8w7"
Oct 30 22:15:57.238291 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.238120    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vql85\" (UniqueName: \"kubernetes.io/projected/c7adcbc6-56d2-4e03-88b0-57ef2cae32d4-kube-api-access-vql85\") pod \"model-serving-api-56ddc4c448-f6qxz\" (UID: \"c7adcbc6-56d2-4e03-88b0-57ef2cae32d4\") " pod="kserve/model-serving-api-56ddc4c448-f6qxz"
Oct 30 22:15:57.262757 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.262727    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-6m8w7"
Oct 30 22:15:57.515440 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.515364    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-f6qxz"
Oct 30 22:15:57.633303 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.633270    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-6m8w7"]
Oct 30 22:15:57.636216 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:15:57.636192    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe12cdf3_ae3b_469b_a303_dd178829d1e5.slice/crio-c5cdb7b1d119731459c4861beb30ca961701d8c67f9ffb4e9eaeb4bcabd24552 WatchSource:0}: Error finding container c5cdb7b1d119731459c4861beb30ca961701d8c67f9ffb4e9eaeb4bcabd24552: Status 404 returned error can't find the container with id c5cdb7b1d119731459c4861beb30ca961701d8c67f9ffb4e9eaeb4bcabd24552
Oct 30 22:15:57.888565 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.888527    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-f6qxz"]
Oct 30 22:15:57.891206 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:57.891180    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-6m8w7" event={"ID":"be12cdf3-ae3b-469b-a303-dd178829d1e5","Type":"ContainerStarted","Data":"c5cdb7b1d119731459c4861beb30ca961701d8c67f9ffb4e9eaeb4bcabd24552"}
Oct 30 22:15:57.891850 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:15:57.891828    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7adcbc6_56d2_4e03_88b0_57ef2cae32d4.slice/crio-3a6dfd1b0183319ba239164e4914b57319b340f7fb5fe6ce6a540b98821d341c WatchSource:0}: Error finding container 3a6dfd1b0183319ba239164e4914b57319b340f7fb5fe6ce6a540b98821d341c: Status 404 returned error can't find the container with id 3a6dfd1b0183319ba239164e4914b57319b340f7fb5fe6ce6a540b98821d341c
Oct 30 22:15:58.896273 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:15:58.896238    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-f6qxz" event={"ID":"c7adcbc6-56d2-4e03-88b0-57ef2cae32d4","Type":"ContainerStarted","Data":"3a6dfd1b0183319ba239164e4914b57319b340f7fb5fe6ce6a540b98821d341c"}
Oct 30 22:15:59.064999 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:15:59.064950    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:16:00.906616 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:16:00.906577    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-6m8w7" event={"ID":"be12cdf3-ae3b-469b-a303-dd178829d1e5","Type":"ContainerStarted","Data":"47668f24bf292d6f2b0ce685239d586640f91d6465c960e14315014e27035711"}
Oct 30 22:16:00.907009 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:16:00.906679    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7f9f8cdc65-6m8w7"
Oct 30 22:16:00.908241 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:16:00.908213    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-f6qxz" event={"ID":"c7adcbc6-56d2-4e03-88b0-57ef2cae32d4","Type":"ContainerStarted","Data":"762ff7f4624f1f764c976a1d4f9165f1256c8529b8f08c6ab9366ee6b8f7b1ed"}
Oct 30 22:16:00.908383 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:16:00.908363    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-56ddc4c448-f6qxz"
Oct 30 22:16:01.026011 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:16:01.025638    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/odh-model-controller-7f9f8cdc65-6m8w7" podStartSLOduration=1.832693641 podStartE2EDuration="5.025621643s" totalImagesPullingTime="3.192928002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:01.025144177 +0000 UTC m=+459.024491740" watchObservedRunningTime="2026-10-30 22:16:01.025621643 +0000 UTC m=+459.024969200"
Oct 30 22:16:01.144119 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:16:01.144015    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/model-serving-api-56ddc4c448-f6qxz" podStartSLOduration=2.894717297 podStartE2EDuration="5.143999951s" totalImagesPullingTime="2.249282654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:01.143287318 +0000 UTC m=+459.142634881" watchObservedRunningTime="2026-10-30 22:16:01.143999951 +0000 UTC m=+459.143347513"
Oct 30 22:16:11.914864 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:16:11.914826    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7f9f8cdc65-6m8w7"
Oct 30 22:16:11.916681 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:16:11.916663    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-56ddc4c448-f6qxz"
Oct 30 22:16:29.047931 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:16:29.047895    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:16:59.048536 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:16:59.048453    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:17:29.048367 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:17:29.048316    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:17:59.048461 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:17:59.048414    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:18:18.108667 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:18.108631    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-gjrsc"]
Oct 30 22:18:18.110850 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:18.110834    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-5c74974785-gjrsc"
Oct 30 22:18:18.164888 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:18.164856    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Oct 30 22:18:18.213384 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:18.213344    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-vf6ms\""
Oct 30 22:18:18.213384 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:18.213359    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Oct 30 22:18:18.214424 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:18.214404    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-gjrsc"]
Oct 30 22:18:18.216635 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:18.216618    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfvwb\" (UniqueName: \"kubernetes.io/projected/bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff-kube-api-access-rfvwb\") pod \"seaweedfs-tls-serving-5c74974785-gjrsc\" (UID: \"bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff\") " pod="kserve/seaweedfs-tls-serving-5c74974785-gjrsc"
Oct 30 22:18:18.216702 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:18.216656    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff-data\") pod \"seaweedfs-tls-serving-5c74974785-gjrsc\" (UID: \"bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff\") " pod="kserve/seaweedfs-tls-serving-5c74974785-gjrsc"
Oct 30 22:18:18.216828 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:18.216806    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-gjrsc\" (UID: \"bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff\") " pod="kserve/seaweedfs-tls-serving-5c74974785-gjrsc"
Oct 30 22:18:18.317372 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:18.317320    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfvwb\" (UniqueName: \"kubernetes.io/projected/bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff-kube-api-access-rfvwb\") pod \"seaweedfs-tls-serving-5c74974785-gjrsc\" (UID: \"bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff\") " pod="kserve/seaweedfs-tls-serving-5c74974785-gjrsc"
Oct 30 22:18:18.317551 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:18.317390    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff-data\") pod \"seaweedfs-tls-serving-5c74974785-gjrsc\" (UID: \"bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff\") " pod="kserve/seaweedfs-tls-serving-5c74974785-gjrsc"
Oct 30 22:18:18.317551 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:18.317476    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-gjrsc\" (UID: \"bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff\") " pod="kserve/seaweedfs-tls-serving-5c74974785-gjrsc"
Oct 30 22:18:18.317770 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:18.317749    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff-data\") pod \"seaweedfs-tls-serving-5c74974785-gjrsc\" (UID: \"bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff\") " pod="kserve/seaweedfs-tls-serving-5c74974785-gjrsc"
Oct 30 22:18:18.319964 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:18.319940    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-gjrsc\" (UID: \"bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff\") " pod="kserve/seaweedfs-tls-serving-5c74974785-gjrsc"
Oct 30 22:18:18.380547 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:18.380480    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfvwb\" (UniqueName: \"kubernetes.io/projected/bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff-kube-api-access-rfvwb\") pod \"seaweedfs-tls-serving-5c74974785-gjrsc\" (UID: \"bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff\") " pod="kserve/seaweedfs-tls-serving-5c74974785-gjrsc"
Oct 30 22:18:18.420443 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:18.420410    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-5c74974785-gjrsc"
Oct 30 22:18:18.791375 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:18.791346    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-gjrsc"]
Oct 30 22:18:18.794838 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:18:18.794813    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd2efc6_4f72_44ac_a6b1_c5c6e2eeccff.slice/crio-993bc499b5baf55ef2f7d3c2fc510e901347f8b2c75b1d640d5cf0cb92c30a8e WatchSource:0}: Error finding container 993bc499b5baf55ef2f7d3c2fc510e901347f8b2c75b1d640d5cf0cb92c30a8e: Status 404 returned error can't find the container with id 993bc499b5baf55ef2f7d3c2fc510e901347f8b2c75b1d640d5cf0cb92c30a8e
Oct 30 22:18:19.441501 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:19.441461    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-5c74974785-gjrsc" event={"ID":"bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff","Type":"ContainerStarted","Data":"993bc499b5baf55ef2f7d3c2fc510e901347f8b2c75b1d640d5cf0cb92c30a8e"}
Oct 30 22:18:22.190426 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:22.190385    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:18:22.192347 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:22.192309    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:18:22.457052 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:22.456956    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-5c74974785-gjrsc" event={"ID":"bcd2efc6-4f72-44ac-a6b1-c5c6e2eeccff","Type":"ContainerStarted","Data":"d006499b3035a1cc596266d4d27b67cbb454bc88f7a56c37ab59ff7dcbceef7d"}
Oct 30 22:18:22.574239 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:22.574184    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-5c74974785-gjrsc" podStartSLOduration=1.704056516 podStartE2EDuration="4.574168688s" totalImagesPullingTime="2.870112172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:22.573670132 +0000 UTC m=+600.573017695" watchObservedRunningTime="2026-10-30 22:18:22.574168688 +0000 UTC m=+600.573516251"
Oct 30 22:18:29.048756 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:18:29.048716    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:18:58.398018 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.397985    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"]
Oct 30 22:18:58.405969 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.405928    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:18:58.454432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.454390    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c211f82-f030-40ba-8733-47dd2501efc3-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:18:58.454432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.454434    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c211f82-f030-40ba-8733-47dd2501efc3-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:18:58.454638 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.454458    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c211f82-f030-40ba-8733-47dd2501efc3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:18:58.454638 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.454522    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c211f82-f030-40ba-8733-47dd2501efc3-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:18:58.454638 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.454581    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhjhm\" (UniqueName: \"kubernetes.io/projected/9c211f82-f030-40ba-8733-47dd2501efc3-kube-api-access-vhjhm\") pod \"isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:18:58.460263 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.460243    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"seaweedfs-s3-creds\""
Oct 30 22:18:58.460324 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.460243    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"default-dockercfg-s9cqw\""
Oct 30 22:18:58.508956 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.508921    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Oct 30 22:18:58.509238 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.509213    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Oct 30 22:18:58.509448 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.509247    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:18:58.509448 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.509320    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"openshift-service-ca.crt\""
Oct 30 22:18:58.509448 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.509357    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"kube-root-ca.crt\""
Oct 30 22:18:58.511165 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.511144    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"]
Oct 30 22:18:58.555265 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.555237    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhjhm\" (UniqueName: \"kubernetes.io/projected/9c211f82-f030-40ba-8733-47dd2501efc3-kube-api-access-vhjhm\") pod \"isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:18:58.555461 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.555272    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c211f82-f030-40ba-8733-47dd2501efc3-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:18:58.555461 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.555297    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c211f82-f030-40ba-8733-47dd2501efc3-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:18:58.555461 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.555359    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c211f82-f030-40ba-8733-47dd2501efc3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:18:58.555461 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.555387    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c211f82-f030-40ba-8733-47dd2501efc3-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:18:58.555797 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.555777    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c211f82-f030-40ba-8733-47dd2501efc3-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:18:58.556043 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.556018    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c211f82-f030-40ba-8733-47dd2501efc3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:18:58.556151 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.556078    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c211f82-f030-40ba-8733-47dd2501efc3-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:18:58.557992 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.557972    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c211f82-f030-40ba-8733-47dd2501efc3-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:18:58.618171 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.618130    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhjhm\" (UniqueName: \"kubernetes.io/projected/9c211f82-f030-40ba-8733-47dd2501efc3-kube-api-access-vhjhm\") pod \"isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:18:58.726506 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:58.726415    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:18:59.048900 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:18:59.048863    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:18:59.157222 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:59.157040    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"]
Oct 30 22:18:59.160025 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:18:59.159993    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c211f82_f030_40ba_8733_47dd2501efc3.slice/crio-707c4179535d022adbc03129c52bd2c2cc14fa053e90d953d898575c4b0be659 WatchSource:0}: Error finding container 707c4179535d022adbc03129c52bd2c2cc14fa053e90d953d898575c4b0be659: Status 404 returned error can't find the container with id 707c4179535d022adbc03129c52bd2c2cc14fa053e90d953d898575c4b0be659
Oct 30 22:18:59.607671 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:18:59.607628    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" event={"ID":"9c211f82-f030-40ba-8733-47dd2501efc3","Type":"ContainerStarted","Data":"707c4179535d022adbc03129c52bd2c2cc14fa053e90d953d898575c4b0be659"}
Oct 30 22:19:03.627663 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:03.627622    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" event={"ID":"9c211f82-f030-40ba-8733-47dd2501efc3","Type":"ContainerStarted","Data":"27af75dbf14b4960148cfa9b8c2c2e714cd2c692e22af02cc94b718e0ab6a3d4"}
Oct 30 22:19:07.646120 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:07.646087    4947 generic.go:356] "Generic (PLEG): container finished" podID="9c211f82-f030-40ba-8733-47dd2501efc3" containerID="27af75dbf14b4960148cfa9b8c2c2e714cd2c692e22af02cc94b718e0ab6a3d4" exitCode=0
Oct 30 22:19:07.646560 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:07.646144    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" event={"ID":"9c211f82-f030-40ba-8733-47dd2501efc3","Type":"ContainerDied","Data":"27af75dbf14b4960148cfa9b8c2c2e714cd2c692e22af02cc94b718e0ab6a3d4"}
Oct 30 22:19:29.053305 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:19:29.053262    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:19:30.649286 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:30.649267    4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:19:30.755441 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:30.755409    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" event={"ID":"9c211f82-f030-40ba-8733-47dd2501efc3","Type":"ContainerStarted","Data":"2a6539fccd0b9f7f15c110e1755dfef5776930380f2633af7905a3fb63e5680f"}
Oct 30 22:19:33.771450 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:33.771368    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" event={"ID":"9c211f82-f030-40ba-8733-47dd2501efc3","Type":"ContainerStarted","Data":"799c1eb2f2a6696274e3efd8f9bca49cae186e9f6bb73aecbd3aa55536367959"}
Oct 30 22:19:42.812247 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:42.812214    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" event={"ID":"9c211f82-f030-40ba-8733-47dd2501efc3","Type":"ContainerStarted","Data":"38e229d0502e8f885dd75d65d4e605493e84d2d480c6ec75a59d5430c14a3d43"}
Oct 30 22:19:42.812729 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:42.812545    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:19:42.812729 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:42.812597    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:19:42.814064 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:42.814034    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Oct 30 22:19:42.817620 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:42.817601    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:19:43.035017 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:43.034899    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podStartSLOduration=1.942708691 podStartE2EDuration="45.034878601s" totalImagesPullingTime="39.162167119s" totalInitContainerRuntime="3.930002791s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:58 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:42.985206429 +0000 UTC m=+680.984553995" watchObservedRunningTime="2026-10-30 22:19:43.034878601 +0000 UTC m=+681.034226169"
Oct 30 22:19:43.818135 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:43.818104    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:19:43.818688 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:43.818349    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Oct 30 22:19:43.819303 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:43.819277    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:19:44.823643 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:44.823604    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Oct 30 22:19:44.824077 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:44.823928    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:19:45.827273 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:45.827226    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Oct 30 22:19:45.827731 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:45.827605    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:19:55.827207 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:55.827159    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Oct 30 22:19:55.827823 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:19:55.827603    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:19:59.048261 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:19:59.048228    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:20:05.827179 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:05.827132    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Oct 30 22:20:05.827644 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:05.827531    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:20:15.827689 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:15.827646    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Oct 30 22:20:15.828161 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:15.828129    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:20:25.827360 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:25.827289    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Oct 30 22:20:25.827827 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:25.827769    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:20:29.048098 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:20:29.048053    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:20:35.827862 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:35.827815    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Oct 30 22:20:35.828370 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:35.828309    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:20:45.827865 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:45.827814    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Oct 30 22:20:45.828366 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:45.828346    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:20:55.797563 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:55.797531    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"]
Oct 30 22:20:55.801301 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:55.801283    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:20:55.828574 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:55.828551    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:20:55.828956 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:55.828939    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:20:55.901408 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:55.901370    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-cabundle-cert\") pod \"isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") " pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:20:55.901575 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:55.901519    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kserve-provision-location\") pod \"isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") " pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:20:55.901676 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:55.901660    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlj9h\" (UniqueName: \"kubernetes.io/projected/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kube-api-access-jlj9h\") pod \"isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") " pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:20:55.901712 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:55.901693    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kube-rbac-proxy-sar-config\") pod \"isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") " pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:20:55.901824 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:55.901805    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-proxy-tls\") pod \"isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") " pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:20:55.909280 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:55.909257    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:20:55.909280 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:55.909257    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-primary-921a4c-kube-rbac-proxy-sar-config\""
Oct 30 22:20:55.909475 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:55.909265    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:20:55.909475 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:55.909302    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:20:55.909475 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:55.909322    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:20:55.909475 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:55.909352    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:20:55.909475 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:55.909376    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-primary-921a4c-predictor-serving-cert\""
Oct 30 22:20:55.911112 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:55.911093    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"]
Oct 30 22:20:56.002508 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:56.002481    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kserve-provision-location\") pod \"isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") " pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:20:56.002677 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:56.002520    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlj9h\" (UniqueName: \"kubernetes.io/projected/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kube-api-access-jlj9h\") pod \"isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") " pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:20:56.002677 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:56.002539    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kube-rbac-proxy-sar-config\") pod \"isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") " pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:20:56.002677 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:56.002657    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-proxy-tls\") pod \"isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") " pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:20:56.002844 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:56.002718    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-cabundle-cert\") pod \"isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") " pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:20:56.002938 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:56.002917    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kserve-provision-location\") pod \"isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") " pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:20:56.003366 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:56.003295    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kube-rbac-proxy-sar-config\") pod \"isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") " pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:20:56.003576 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:56.003425    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-cabundle-cert\") pod \"isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") " pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:20:56.005156 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:56.005136    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-proxy-tls\") pod \"isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") " pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:20:56.065119 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:56.065057    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlj9h\" (UniqueName: \"kubernetes.io/projected/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kube-api-access-jlj9h\") pod \"isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") " pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:20:56.118687 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:56.118660    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:20:56.548082 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:56.548058    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"]
Oct 30 22:20:56.550932 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:20:56.550900    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6489abac_6c1b_4f5f_8b02_0a95fd735ee7.slice/crio-9dbd03175ae7bdde0f362a1d3cbbe2800b8ce689192b840a40c95c49714409d1 WatchSource:0}: Error finding container 9dbd03175ae7bdde0f362a1d3cbbe2800b8ce689192b840a40c95c49714409d1: Status 404 returned error can't find the container with id 9dbd03175ae7bdde0f362a1d3cbbe2800b8ce689192b840a40c95c49714409d1
Oct 30 22:20:57.112599 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:57.112556    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" event={"ID":"6489abac-6c1b-4f5f-8b02-0a95fd735ee7","Type":"ContainerStarted","Data":"fa3f0b19e7554e4e41bbbf44ae06eb29df87ccb9181a054ad6526023c10ff01f"}
Oct 30 22:20:57.112968 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:20:57.112638    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" event={"ID":"6489abac-6c1b-4f5f-8b02-0a95fd735ee7","Type":"ContainerStarted","Data":"9dbd03175ae7bdde0f362a1d3cbbe2800b8ce689192b840a40c95c49714409d1"}
Oct 30 22:20:59.047839 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:20:59.047803    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:21:01.130727 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:01.130694    4947 generic.go:356] "Generic (PLEG): container finished" podID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerID="fa3f0b19e7554e4e41bbbf44ae06eb29df87ccb9181a054ad6526023c10ff01f" exitCode=0
Oct 30 22:21:01.131156 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:01.130769    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" event={"ID":"6489abac-6c1b-4f5f-8b02-0a95fd735ee7","Type":"ContainerDied","Data":"fa3f0b19e7554e4e41bbbf44ae06eb29df87ccb9181a054ad6526023c10ff01f"}
Oct 30 22:21:02.137318 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:02.137237    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" event={"ID":"6489abac-6c1b-4f5f-8b02-0a95fd735ee7","Type":"ContainerStarted","Data":"7b2ffe16515b5152aac651b51317d64770be5caa3aa613c6bc144d13ad8e2bba"}
Oct 30 22:21:02.137318 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:02.137308    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" event={"ID":"6489abac-6c1b-4f5f-8b02-0a95fd735ee7","Type":"ContainerStarted","Data":"c6a7a2408373a14dfb78838b01006312c37c059bc2143937c903ec146be4fb30"}
Oct 30 22:21:02.137864 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:02.137530    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:21:02.262158 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:02.262105    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" podStartSLOduration=3.322766956 podStartE2EDuration="7.262092433s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.939325477s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:02.261554116 +0000 UTC m=+760.260901689" watchObservedRunningTime="2026-10-30 22:21:02.262092433 +0000 UTC m=+760.261439993"
Oct 30 22:21:03.141885 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:03.141855    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:21:03.143144 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:03.143115    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 30 22:21:04.151425 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:04.151380    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 30 22:21:09.155862 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:09.155832    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:21:09.156291 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:09.156266    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 30 22:21:12.483754 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:12.483720    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"]
Oct 30 22:21:12.484177 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:12.483981    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kserve-container" containerID="cri-o://2a6539fccd0b9f7f15c110e1755dfef5776930380f2633af7905a3fb63e5680f" gracePeriod=30
Oct 30 22:21:12.484177 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:12.484024    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kube-rbac-proxy" containerID="cri-o://799c1eb2f2a6696274e3efd8f9bca49cae186e9f6bb73aecbd3aa55536367959" gracePeriod=30
Oct 30 22:21:12.484177 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:12.484032    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="agent" containerID="cri-o://38e229d0502e8f885dd75d65d4e605493e84d2d480c6ec75a59d5430c14a3d43" gracePeriod=30
Oct 30 22:21:12.813047 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:12.813008    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Oct 30 22:21:13.194138 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:13.194055    4947 generic.go:356] "Generic (PLEG): container finished" podID="9c211f82-f030-40ba-8733-47dd2501efc3" containerID="799c1eb2f2a6696274e3efd8f9bca49cae186e9f6bb73aecbd3aa55536367959" exitCode=0
Oct 30 22:21:13.194138 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:13.194125    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" event={"ID":"9c211f82-f030-40ba-8733-47dd2501efc3","Type":"ContainerDied","Data":"799c1eb2f2a6696274e3efd8f9bca49cae186e9f6bb73aecbd3aa55536367959"}
Oct 30 22:21:15.827827 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:15.827792    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:21:15.829005 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:15.828979    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Oct 30 22:21:17.812917 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:17.812878    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Oct 30 22:21:18.219805 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:18.219724    4947 generic.go:356] "Generic (PLEG): container finished" podID="9c211f82-f030-40ba-8733-47dd2501efc3" containerID="2a6539fccd0b9f7f15c110e1755dfef5776930380f2633af7905a3fb63e5680f" exitCode=0
Oct 30 22:21:18.219805 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:18.219776    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" event={"ID":"9c211f82-f030-40ba-8733-47dd2501efc3","Type":"ContainerDied","Data":"2a6539fccd0b9f7f15c110e1755dfef5776930380f2633af7905a3fb63e5680f"}
Oct 30 22:21:19.156428 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:19.156383    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 30 22:21:22.813146 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:22.813110    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Oct 30 22:21:22.813543 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:22.813238    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:21:25.827739 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:25.827697    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:21:25.828960 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:25.828931    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Oct 30 22:21:27.813035 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:27.812991    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Oct 30 22:21:29.048637 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:21:29.048599    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:21:29.157050 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:29.157014    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 30 22:21:32.813117 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:32.813080    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Oct 30 22:21:35.827955 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:35.827917    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:21:35.828413 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:35.828054    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:21:35.829113 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:35.829093    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Oct 30 22:21:35.829192 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:35.829181    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:21:37.813806 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:37.813767    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Oct 30 22:21:39.157042 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:39.157005    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 30 22:21:42.626055 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:42.626032    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:21:42.678971 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:42.678937    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c211f82-f030-40ba-8733-47dd2501efc3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c211f82-f030-40ba-8733-47dd2501efc3-cabundle-cert\") pod \"9c211f82-f030-40ba-8733-47dd2501efc3\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") "
Oct 30 22:21:42.679109 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:42.678992    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c211f82-f030-40ba-8733-47dd2501efc3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c211f82-f030-40ba-8733-47dd2501efc3-kube-rbac-proxy-sar-config\") pod \"9c211f82-f030-40ba-8733-47dd2501efc3\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") "
Oct 30 22:21:42.679109 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:42.679035    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c211f82-f030-40ba-8733-47dd2501efc3-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c211f82-f030-40ba-8733-47dd2501efc3-proxy-tls\") pod \"9c211f82-f030-40ba-8733-47dd2501efc3\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") "
Oct 30 22:21:42.679109 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:42.679073    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c211f82-f030-40ba-8733-47dd2501efc3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c211f82-f030-40ba-8733-47dd2501efc3-kserve-provision-location\") pod \"9c211f82-f030-40ba-8733-47dd2501efc3\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") "
Oct 30 22:21:42.679229 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:42.679123    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c211f82-f030-40ba-8733-47dd2501efc3-kube-api-access-vhjhm\" (UniqueName: \"kubernetes.io/projected/9c211f82-f030-40ba-8733-47dd2501efc3-kube-api-access-vhjhm\") pod \"9c211f82-f030-40ba-8733-47dd2501efc3\" (UID: \"9c211f82-f030-40ba-8733-47dd2501efc3\") "
Oct 30 22:21:42.679485 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:42.679456    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c211f82-f030-40ba-8733-47dd2501efc3-cabundle-cert" pod "9c211f82-f030-40ba-8733-47dd2501efc3" (UID: "9c211f82-f030-40ba-8733-47dd2501efc3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:42.679485 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:42.679462    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c211f82-f030-40ba-8733-47dd2501efc3-kserve-provision-location" pod "9c211f82-f030-40ba-8733-47dd2501efc3" (UID: "9c211f82-f030-40ba-8733-47dd2501efc3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:42.679622 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:42.679466    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c211f82-f030-40ba-8733-47dd2501efc3-kube-rbac-proxy-sar-config" pod "9c211f82-f030-40ba-8733-47dd2501efc3" (UID: "9c211f82-f030-40ba-8733-47dd2501efc3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:42.681416 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:42.681390    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c211f82-f030-40ba-8733-47dd2501efc3-kube-api-access-vhjhm" pod "9c211f82-f030-40ba-8733-47dd2501efc3" (UID: "9c211f82-f030-40ba-8733-47dd2501efc3"). InnerVolumeSpecName "kube-api-access-vhjhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:42.681493 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:42.681470    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c211f82-f030-40ba-8733-47dd2501efc3-proxy-tls" pod "9c211f82-f030-40ba-8733-47dd2501efc3" (UID: "9c211f82-f030-40ba-8733-47dd2501efc3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:42.780436 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:42.780357    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhjhm\" (UniqueName: \"kubernetes.io/projected/9c211f82-f030-40ba-8733-47dd2501efc3-kube-api-access-vhjhm\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:42.780436 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:42.780384    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c211f82-f030-40ba-8733-47dd2501efc3-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:42.780436 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:42.780400    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c211f82-f030-40ba-8733-47dd2501efc3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:42.780436 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:42.780410    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c211f82-f030-40ba-8733-47dd2501efc3-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:42.780436 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:42.780418    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c211f82-f030-40ba-8733-47dd2501efc3-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:43.321580 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.321545    4947 generic.go:356] "Generic (PLEG): container finished" podID="9c211f82-f030-40ba-8733-47dd2501efc3" containerID="38e229d0502e8f885dd75d65d4e605493e84d2d480c6ec75a59d5430c14a3d43" exitCode=0
Oct 30 22:21:43.321776 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.321604    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"
Oct 30 22:21:43.321776 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.321619    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" event={"ID":"9c211f82-f030-40ba-8733-47dd2501efc3","Type":"ContainerDied","Data":"38e229d0502e8f885dd75d65d4e605493e84d2d480c6ec75a59d5430c14a3d43"}
Oct 30 22:21:43.321776 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.321697    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5" event={"ID":"9c211f82-f030-40ba-8733-47dd2501efc3","Type":"ContainerDied","Data":"707c4179535d022adbc03129c52bd2c2cc14fa053e90d953d898575c4b0be659"}
Oct 30 22:21:43.321776 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.321739    4947 scope.go:122] "RemoveContainer" containerID="38e229d0502e8f885dd75d65d4e605493e84d2d480c6ec75a59d5430c14a3d43"
Oct 30 22:21:43.331092 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.331076    4947 scope.go:122] "RemoveContainer" containerID="799c1eb2f2a6696274e3efd8f9bca49cae186e9f6bb73aecbd3aa55536367959"
Oct 30 22:21:43.338229 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.338213    4947 scope.go:122] "RemoveContainer" containerID="2a6539fccd0b9f7f15c110e1755dfef5776930380f2633af7905a3fb63e5680f"
Oct 30 22:21:43.347408 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.347388    4947 scope.go:122] "RemoveContainer" containerID="27af75dbf14b4960148cfa9b8c2c2e714cd2c692e22af02cc94b718e0ab6a3d4"
Oct 30 22:21:43.354960 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.354946    4947 scope.go:122] "RemoveContainer" containerID="38e229d0502e8f885dd75d65d4e605493e84d2d480c6ec75a59d5430c14a3d43"
Oct 30 22:21:43.355188 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:21:43.355169    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e229d0502e8f885dd75d65d4e605493e84d2d480c6ec75a59d5430c14a3d43\": container with ID starting with 38e229d0502e8f885dd75d65d4e605493e84d2d480c6ec75a59d5430c14a3d43 not found: ID does not exist" containerID="38e229d0502e8f885dd75d65d4e605493e84d2d480c6ec75a59d5430c14a3d43"
Oct 30 22:21:43.355245 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.355202    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e229d0502e8f885dd75d65d4e605493e84d2d480c6ec75a59d5430c14a3d43"} err="failed to get container status \"38e229d0502e8f885dd75d65d4e605493e84d2d480c6ec75a59d5430c14a3d43\": rpc error: code = NotFound desc = could not find container \"38e229d0502e8f885dd75d65d4e605493e84d2d480c6ec75a59d5430c14a3d43\": container with ID starting with 38e229d0502e8f885dd75d65d4e605493e84d2d480c6ec75a59d5430c14a3d43 not found: ID does not exist"
Oct 30 22:21:43.355245 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.355224    4947 scope.go:122] "RemoveContainer" containerID="799c1eb2f2a6696274e3efd8f9bca49cae186e9f6bb73aecbd3aa55536367959"
Oct 30 22:21:43.355535 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:21:43.355519    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799c1eb2f2a6696274e3efd8f9bca49cae186e9f6bb73aecbd3aa55536367959\": container with ID starting with 799c1eb2f2a6696274e3efd8f9bca49cae186e9f6bb73aecbd3aa55536367959 not found: ID does not exist" containerID="799c1eb2f2a6696274e3efd8f9bca49cae186e9f6bb73aecbd3aa55536367959"
Oct 30 22:21:43.355585 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.355542    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799c1eb2f2a6696274e3efd8f9bca49cae186e9f6bb73aecbd3aa55536367959"} err="failed to get container status \"799c1eb2f2a6696274e3efd8f9bca49cae186e9f6bb73aecbd3aa55536367959\": rpc error: code = NotFound desc = could not find container \"799c1eb2f2a6696274e3efd8f9bca49cae186e9f6bb73aecbd3aa55536367959\": container with ID starting with 799c1eb2f2a6696274e3efd8f9bca49cae186e9f6bb73aecbd3aa55536367959 not found: ID does not exist"
Oct 30 22:21:43.355585 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.355558    4947 scope.go:122] "RemoveContainer" containerID="2a6539fccd0b9f7f15c110e1755dfef5776930380f2633af7905a3fb63e5680f"
Oct 30 22:21:43.355810 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:21:43.355793    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6539fccd0b9f7f15c110e1755dfef5776930380f2633af7905a3fb63e5680f\": container with ID starting with 2a6539fccd0b9f7f15c110e1755dfef5776930380f2633af7905a3fb63e5680f not found: ID does not exist" containerID="2a6539fccd0b9f7f15c110e1755dfef5776930380f2633af7905a3fb63e5680f"
Oct 30 22:21:43.355853 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.355816    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6539fccd0b9f7f15c110e1755dfef5776930380f2633af7905a3fb63e5680f"} err="failed to get container status \"2a6539fccd0b9f7f15c110e1755dfef5776930380f2633af7905a3fb63e5680f\": rpc error: code = NotFound desc = could not find container \"2a6539fccd0b9f7f15c110e1755dfef5776930380f2633af7905a3fb63e5680f\": container with ID starting with 2a6539fccd0b9f7f15c110e1755dfef5776930380f2633af7905a3fb63e5680f not found: ID does not exist"
Oct 30 22:21:43.355853 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.355829    4947 scope.go:122] "RemoveContainer" containerID="27af75dbf14b4960148cfa9b8c2c2e714cd2c692e22af02cc94b718e0ab6a3d4"
Oct 30 22:21:43.356042 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:21:43.356023    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27af75dbf14b4960148cfa9b8c2c2e714cd2c692e22af02cc94b718e0ab6a3d4\": container with ID starting with 27af75dbf14b4960148cfa9b8c2c2e714cd2c692e22af02cc94b718e0ab6a3d4 not found: ID does not exist" containerID="27af75dbf14b4960148cfa9b8c2c2e714cd2c692e22af02cc94b718e0ab6a3d4"
Oct 30 22:21:43.356084 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.356048    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27af75dbf14b4960148cfa9b8c2c2e714cd2c692e22af02cc94b718e0ab6a3d4"} err="failed to get container status \"27af75dbf14b4960148cfa9b8c2c2e714cd2c692e22af02cc94b718e0ab6a3d4\": rpc error: code = NotFound desc = could not find container \"27af75dbf14b4960148cfa9b8c2c2e714cd2c692e22af02cc94b718e0ab6a3d4\": container with ID starting with 27af75dbf14b4960148cfa9b8c2c2e714cd2c692e22af02cc94b718e0ab6a3d4 not found: ID does not exist"
Oct 30 22:21:43.539167 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.539137    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"]
Oct 30 22:21:43.589173 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:43.589090    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-sklearn-batcher-predictor-7b6c58ff65-9cqm5"]
Oct 30 22:21:44.158769 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:44.158734    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" path="/var/lib/kubelet/pods/9c211f82-f030-40ba-8733-47dd2501efc3/volumes"
Oct 30 22:21:46.015681 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.015645    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"]
Oct 30 22:21:46.016071 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.015984    4947 cpu_manager.go:414] "containerMap: removing container" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="agent"
Oct 30 22:21:46.016071 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.015995    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="agent"
Oct 30 22:21:46.016071 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.016007    4947 cpu_manager.go:414] "containerMap: removing container" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kserve-container"
Oct 30 22:21:46.016071 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.016012    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kserve-container"
Oct 30 22:21:46.016071 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.016034    4947 cpu_manager.go:414] "containerMap: removing container" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="storage-initializer"
Oct 30 22:21:46.016071 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.016039    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="storage-initializer"
Oct 30 22:21:46.016071 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.016048    4947 cpu_manager.go:414] "containerMap: removing container" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kube-rbac-proxy"
Oct 30 22:21:46.016071 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.016053    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kube-rbac-proxy"
Oct 30 22:21:46.016299 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.016098    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kube-rbac-proxy"
Oct 30 22:21:46.016299 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.016107    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="agent"
Oct 30 22:21:46.016299 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.016113    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c211f82-f030-40ba-8733-47dd2501efc3" containerName="kserve-container"
Oct 30 22:21:46.020978 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.020959    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:46.074976 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.074943    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"openshift-service-ca.crt\""
Oct 30 22:21:46.074976 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.074949    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:21:46.075151 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.074949    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"kube-root-ca.crt\""
Oct 30 22:21:46.107684 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.107650    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g5bn\" (UniqueName: \"kubernetes.io/projected/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kube-api-access-6g5bn\") pod \"isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:46.107818 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.107690    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cd5f470-2251-4f17-a5e5-8fbb823c5244-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:46.107818 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.107716    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:46.107818 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.107791    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:46.107947 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.107839    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cd5f470-2251-4f17-a5e5-8fbb823c5244-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:46.124248 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.124219    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Oct 30 22:21:46.124462 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.124320    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"seaweedfs-s3-creds\""
Oct 30 22:21:46.124526 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.124385    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"default-dockercfg-s9cqw\""
Oct 30 22:21:46.124526 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.124413    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Oct 30 22:21:46.126397 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.126373    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"]
Oct 30 22:21:46.208473 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.208439    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g5bn\" (UniqueName: \"kubernetes.io/projected/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kube-api-access-6g5bn\") pod \"isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:46.208667 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.208482    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cd5f470-2251-4f17-a5e5-8fbb823c5244-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:46.208667 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.208520    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:46.208667 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.208545    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:46.208667 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.208575    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cd5f470-2251-4f17-a5e5-8fbb823c5244-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:46.208942 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.208920    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:46.209224 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.209192    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:46.209328 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.209308    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cd5f470-2251-4f17-a5e5-8fbb823c5244-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:46.211090 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.211059    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cd5f470-2251-4f17-a5e5-8fbb823c5244-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:46.288495 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.288463    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g5bn\" (UniqueName: \"kubernetes.io/projected/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kube-api-access-6g5bn\") pod \"isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") " pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:46.339993 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.339959    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:46.767640 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:46.767502    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"]
Oct 30 22:21:46.770418 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:21:46.770387    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd5f470_2251_4f17_a5e5_8fbb823c5244.slice/crio-33d6c6a7d3d19c90a0939d292d051b1500645de1e0eb740c9543ea95b06326a6 WatchSource:0}: Error finding container 33d6c6a7d3d19c90a0939d292d051b1500645de1e0eb740c9543ea95b06326a6: Status 404 returned error can't find the container with id 33d6c6a7d3d19c90a0939d292d051b1500645de1e0eb740c9543ea95b06326a6
Oct 30 22:21:47.341536 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:47.341504    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" event={"ID":"0cd5f470-2251-4f17-a5e5-8fbb823c5244","Type":"ContainerStarted","Data":"c164f1346637c953b11ff3b3e3bfa735a11b83ab6776a84aecac142241dcda5c"}
Oct 30 22:21:47.341924 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:47.341564    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" event={"ID":"0cd5f470-2251-4f17-a5e5-8fbb823c5244","Type":"ContainerStarted","Data":"33d6c6a7d3d19c90a0939d292d051b1500645de1e0eb740c9543ea95b06326a6"}
Oct 30 22:21:49.156832 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:49.156796    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 30 22:21:51.358495 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:51.358467    4947 generic.go:356] "Generic (PLEG): container finished" podID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerID="c164f1346637c953b11ff3b3e3bfa735a11b83ab6776a84aecac142241dcda5c" exitCode=0
Oct 30 22:21:51.358875 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:51.358544    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" event={"ID":"0cd5f470-2251-4f17-a5e5-8fbb823c5244","Type":"ContainerDied","Data":"c164f1346637c953b11ff3b3e3bfa735a11b83ab6776a84aecac142241dcda5c"}
Oct 30 22:21:52.364827 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:52.364794    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" event={"ID":"0cd5f470-2251-4f17-a5e5-8fbb823c5244","Type":"ContainerStarted","Data":"021c35b3d9ae7294be1dceb6d5e1302a63de146ddbfa16a91318b29f22de8793"}
Oct 30 22:21:52.365244 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:52.364869    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" event={"ID":"0cd5f470-2251-4f17-a5e5-8fbb823c5244","Type":"ContainerStarted","Data":"57f58cd1a836ead9342263a5d6b974637bf8c849522c50dee2c9c689e9bab97d"}
Oct 30 22:21:52.365244 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:52.364894    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" event={"ID":"0cd5f470-2251-4f17-a5e5-8fbb823c5244","Type":"ContainerStarted","Data":"cc339937956ab7eae85295365cfeb930cf28b22dbe73479821aa5380c293360f"}
Oct 30 22:21:52.365244 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:52.365169    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:52.365404 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:52.365306    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:52.366568 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:52.366545    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:5000: connect: connection refused"
Oct 30 22:21:52.487142 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:52.487085    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podStartSLOduration=3.292991407 podStartE2EDuration="7.487063986s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.194072579s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:52.486832527 +0000 UTC m=+810.486180092" watchObservedRunningTime="2026-10-30 22:21:52.487063986 +0000 UTC m=+810.486411546"
Oct 30 22:21:53.369428 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:53.369396    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:53.369884 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:53.369553    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:5000: connect: connection refused"
Oct 30 22:21:53.370656 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:53.370633    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:21:54.373538 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:54.373498    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:5000: connect: connection refused"
Oct 30 22:21:54.373966 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:54.373944    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:21:59.048213 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:21:59.048174    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:21:59.156301 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:59.156260    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 30 22:21:59.377833 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:59.377760    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:21:59.378356 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:59.378292    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:5000: connect: connection refused"
Oct 30 22:21:59.378545 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:21:59.378524    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:22:09.156695 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:09.156651    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 30 22:22:09.378301 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:09.378258    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:5000: connect: connection refused"
Oct 30 22:22:09.378777 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:09.378746    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:22:14.153049 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:14.153010    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 30 22:22:19.378714 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:19.378662    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:5000: connect: connection refused"
Oct 30 22:22:19.379221 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:19.379101    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:22:24.158065 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:24.158036    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:22:29.048016 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:22:29.047980    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:22:29.378782 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:29.378692    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:5000: connect: connection refused"
Oct 30 22:22:29.379160 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:29.379137    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:22:39.378366 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:39.378263    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:5000: connect: connection refused"
Oct 30 22:22:39.378943 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:39.378526    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:22:49.378628 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:49.378589    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:5000: connect: connection refused"
Oct 30 22:22:49.379145 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:49.379121    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:22:51.249257 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.249223    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"]
Oct 30 22:22:51.252800 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.252782    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:22:51.313242 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.313218    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"invalid-s3-secret-921a4c\""
Oct 30 22:22:51.334065 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.334038    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2035c390-696e-4873-bcb3-a6e13f656072-kserve-provision-location\") pod \"isvc-secondary-921a4c-predictor-659fd47bd5-trwfk\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") " pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:22:51.334167 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.334074    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2035c390-696e-4873-bcb3-a6e13f656072-cabundle-cert\") pod \"isvc-secondary-921a4c-predictor-659fd47bd5-trwfk\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") " pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:22:51.334210 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.334164    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2035c390-696e-4873-bcb3-a6e13f656072-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-921a4c-predictor-659fd47bd5-trwfk\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") " pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:22:51.334210 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.334201    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jtfj\" (UniqueName: \"kubernetes.io/projected/2035c390-696e-4873-bcb3-a6e13f656072-kube-api-access-7jtfj\") pod \"isvc-secondary-921a4c-predictor-659fd47bd5-trwfk\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") " pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:22:51.334276 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.334240    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2035c390-696e-4873-bcb3-a6e13f656072-proxy-tls\") pod \"isvc-secondary-921a4c-predictor-659fd47bd5-trwfk\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") " pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:22:51.362240 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.362214    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"invalid-s3-sa-921a4c-dockercfg-vxh5q\""
Oct 30 22:22:51.362240 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.362224    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-secondary-921a4c-predictor-serving-cert\""
Oct 30 22:22:51.362455 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.362263    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-secondary-921a4c-kube-rbac-proxy-sar-config\""
Oct 30 22:22:51.363733 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.363712    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"]
Oct 30 22:22:51.435529 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.435505    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2035c390-696e-4873-bcb3-a6e13f656072-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-921a4c-predictor-659fd47bd5-trwfk\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") " pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:22:51.435680 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.435534    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtfj\" (UniqueName: \"kubernetes.io/projected/2035c390-696e-4873-bcb3-a6e13f656072-kube-api-access-7jtfj\") pod \"isvc-secondary-921a4c-predictor-659fd47bd5-trwfk\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") " pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:22:51.435680 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.435559    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2035c390-696e-4873-bcb3-a6e13f656072-proxy-tls\") pod \"isvc-secondary-921a4c-predictor-659fd47bd5-trwfk\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") " pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:22:51.435680 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.435591    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2035c390-696e-4873-bcb3-a6e13f656072-kserve-provision-location\") pod \"isvc-secondary-921a4c-predictor-659fd47bd5-trwfk\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") " pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:22:51.435680 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.435611    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2035c390-696e-4873-bcb3-a6e13f656072-cabundle-cert\") pod \"isvc-secondary-921a4c-predictor-659fd47bd5-trwfk\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") " pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:22:51.436000 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.435981    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2035c390-696e-4873-bcb3-a6e13f656072-kserve-provision-location\") pod \"isvc-secondary-921a4c-predictor-659fd47bd5-trwfk\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") " pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:22:51.436299 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.436279    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2035c390-696e-4873-bcb3-a6e13f656072-cabundle-cert\") pod \"isvc-secondary-921a4c-predictor-659fd47bd5-trwfk\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") " pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:22:51.436438 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.436419    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2035c390-696e-4873-bcb3-a6e13f656072-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-921a4c-predictor-659fd47bd5-trwfk\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") " pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:22:51.438115 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.438098    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2035c390-696e-4873-bcb3-a6e13f656072-proxy-tls\") pod \"isvc-secondary-921a4c-predictor-659fd47bd5-trwfk\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") " pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:22:51.499305 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.499240    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtfj\" (UniqueName: \"kubernetes.io/projected/2035c390-696e-4873-bcb3-a6e13f656072-kube-api-access-7jtfj\") pod \"isvc-secondary-921a4c-predictor-659fd47bd5-trwfk\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") " pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:22:51.569512 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.569477    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:22:51.997194 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:51.997087    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"]
Oct 30 22:22:52.000190 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:22:52.000164    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2035c390_696e_4873_bcb3_a6e13f656072.slice/crio-7076bb1df0d9a243c27936368ac3e5f881415ac29373ec4546e694996544fc2b WatchSource:0}: Error finding container 7076bb1df0d9a243c27936368ac3e5f881415ac29373ec4546e694996544fc2b: Status 404 returned error can't find the container with id 7076bb1df0d9a243c27936368ac3e5f881415ac29373ec4546e694996544fc2b
Oct 30 22:22:52.612225 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:52.612182    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk" event={"ID":"2035c390-696e-4873-bcb3-a6e13f656072","Type":"ContainerStarted","Data":"86fcb329b28a6ce4b6fd5920d8aa01086675aae9850d521e2f9dd7d07473a47d"}
Oct 30 22:22:52.612637 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:52.612265    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk" event={"ID":"2035c390-696e-4873-bcb3-a6e13f656072","Type":"ContainerStarted","Data":"7076bb1df0d9a243c27936368ac3e5f881415ac29373ec4546e694996544fc2b"}
Oct 30 22:22:58.637911 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:58.637884    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-secondary-921a4c-predictor-659fd47bd5-trwfk_2035c390-696e-4873-bcb3-a6e13f656072/storage-initializer/0.log"
Oct 30 22:22:58.638286 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:58.637919    4947 generic.go:356] "Generic (PLEG): container finished" podID="2035c390-696e-4873-bcb3-a6e13f656072" containerID="86fcb329b28a6ce4b6fd5920d8aa01086675aae9850d521e2f9dd7d07473a47d" exitCode=1
Oct 30 22:22:58.638286 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:58.637987    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk" event={"ID":"2035c390-696e-4873-bcb3-a6e13f656072","Type":"ContainerDied","Data":"86fcb329b28a6ce4b6fd5920d8aa01086675aae9850d521e2f9dd7d07473a47d"}
Oct 30 22:22:59.048743 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:22:59.048709    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:22:59.378255 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:59.378165    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:5000: connect: connection refused"
Oct 30 22:22:59.378678 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:59.378650    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:22:59.643744 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:59.643665    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-secondary-921a4c-predictor-659fd47bd5-trwfk_2035c390-696e-4873-bcb3-a6e13f656072/storage-initializer/0.log"
Oct 30 22:22:59.644125 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:22:59.643784    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk" event={"ID":"2035c390-696e-4873-bcb3-a6e13f656072","Type":"ContainerStarted","Data":"d49e8dfdb6ea9ee9bc0906637ed58af760308146dc9ac2beddbde5aaecef7b01"}
Oct 30 22:23:03.662499 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:03.662465    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-secondary-921a4c-predictor-659fd47bd5-trwfk_2035c390-696e-4873-bcb3-a6e13f656072/storage-initializer/1.log"
Oct 30 22:23:03.662920 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:03.662883    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-secondary-921a4c-predictor-659fd47bd5-trwfk_2035c390-696e-4873-bcb3-a6e13f656072/storage-initializer/0.log"
Oct 30 22:23:03.663015 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:03.662914    4947 generic.go:356] "Generic (PLEG): container finished" podID="2035c390-696e-4873-bcb3-a6e13f656072" containerID="d49e8dfdb6ea9ee9bc0906637ed58af760308146dc9ac2beddbde5aaecef7b01" exitCode=1
Oct 30 22:23:03.663015 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:03.662986    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk" event={"ID":"2035c390-696e-4873-bcb3-a6e13f656072","Type":"ContainerDied","Data":"d49e8dfdb6ea9ee9bc0906637ed58af760308146dc9ac2beddbde5aaecef7b01"}
Oct 30 22:23:03.663093 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:03.663054    4947 scope.go:122] "RemoveContainer" containerID="86fcb329b28a6ce4b6fd5920d8aa01086675aae9850d521e2f9dd7d07473a47d"
Oct 30 22:23:03.663383 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:03.663366    4947 scope.go:122] "RemoveContainer" containerID="86fcb329b28a6ce4b6fd5920d8aa01086675aae9850d521e2f9dd7d07473a47d"
Oct 30 22:23:03.673918 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:23:03.673895    4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-921a4c-predictor-659fd47bd5-trwfk_e2e-gw1-bc5112d610(2035c390-696e-4873-bcb3-a6e13f656072)\"" pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk" podUID="2035c390-696e-4873-bcb3-a6e13f656072"
Oct 30 22:23:03.674135 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:23:03.674110    4947 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-921a4c-predictor-659fd47bd5-trwfk_e2e-gw1-bc5112d610_2035c390-696e-4873-bcb3-a6e13f656072_0 in pod sandbox 7076bb1df0d9a243c27936368ac3e5f881415ac29373ec4546e694996544fc2b from index: no such id: '86fcb329b28a6ce4b6fd5920d8aa01086675aae9850d521e2f9dd7d07473a47d'" containerID="86fcb329b28a6ce4b6fd5920d8aa01086675aae9850d521e2f9dd7d07473a47d"
Oct 30 22:23:03.674186 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:03.674145    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86fcb329b28a6ce4b6fd5920d8aa01086675aae9850d521e2f9dd7d07473a47d"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-921a4c-predictor-659fd47bd5-trwfk_e2e-gw1-bc5112d610_2035c390-696e-4873-bcb3-a6e13f656072_0 in pod sandbox 7076bb1df0d9a243c27936368ac3e5f881415ac29373ec4546e694996544fc2b from index: no such id: '86fcb329b28a6ce4b6fd5920d8aa01086675aae9850d521e2f9dd7d07473a47d'"
Oct 30 22:23:04.669486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:04.669462    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-secondary-921a4c-predictor-659fd47bd5-trwfk_2035c390-696e-4873-bcb3-a6e13f656072/storage-initializer/1.log"
Oct 30 22:23:09.378827 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:09.378796    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:23:09.381071 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:09.378938    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:23:13.158507 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.158473    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"]
Oct 30 22:23:13.288929 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.288902    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-secondary-921a4c-predictor-659fd47bd5-trwfk_2035c390-696e-4873-bcb3-a6e13f656072/storage-initializer/1.log"
Oct 30 22:23:13.291395 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.291373    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:23:13.421302 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.421219    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2035c390-696e-4873-bcb3-a6e13f656072-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2035c390-696e-4873-bcb3-a6e13f656072-kube-rbac-proxy-sar-config\") pod \"2035c390-696e-4873-bcb3-a6e13f656072\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") "
Oct 30 22:23:13.421302 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.421283    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2035c390-696e-4873-bcb3-a6e13f656072-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2035c390-696e-4873-bcb3-a6e13f656072-kserve-provision-location\") pod \"2035c390-696e-4873-bcb3-a6e13f656072\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") "
Oct 30 22:23:13.421536 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.421373    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2035c390-696e-4873-bcb3-a6e13f656072-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2035c390-696e-4873-bcb3-a6e13f656072-cabundle-cert\") pod \"2035c390-696e-4873-bcb3-a6e13f656072\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") "
Oct 30 22:23:13.421536 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.421445    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2035c390-696e-4873-bcb3-a6e13f656072-kube-api-access-7jtfj\" (UniqueName: \"kubernetes.io/projected/2035c390-696e-4873-bcb3-a6e13f656072-kube-api-access-7jtfj\") pod \"2035c390-696e-4873-bcb3-a6e13f656072\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") "
Oct 30 22:23:13.421536 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.421474    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2035c390-696e-4873-bcb3-a6e13f656072-proxy-tls\" (UniqueName: \"kubernetes.io/secret/2035c390-696e-4873-bcb3-a6e13f656072-proxy-tls\") pod \"2035c390-696e-4873-bcb3-a6e13f656072\" (UID: \"2035c390-696e-4873-bcb3-a6e13f656072\") "
Oct 30 22:23:13.421787 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.421759    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2035c390-696e-4873-bcb3-a6e13f656072-kserve-provision-location" pod "2035c390-696e-4873-bcb3-a6e13f656072" (UID: "2035c390-696e-4873-bcb3-a6e13f656072"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:13.421945 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.421806    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2035c390-696e-4873-bcb3-a6e13f656072-cabundle-cert" pod "2035c390-696e-4873-bcb3-a6e13f656072" (UID: "2035c390-696e-4873-bcb3-a6e13f656072"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:13.422058 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.421855    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2035c390-696e-4873-bcb3-a6e13f656072-kube-rbac-proxy-sar-config" pod "2035c390-696e-4873-bcb3-a6e13f656072" (UID: "2035c390-696e-4873-bcb3-a6e13f656072"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:13.424209 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.424163    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2035c390-696e-4873-bcb3-a6e13f656072-kube-api-access-7jtfj" pod "2035c390-696e-4873-bcb3-a6e13f656072" (UID: "2035c390-696e-4873-bcb3-a6e13f656072"). InnerVolumeSpecName "kube-api-access-7jtfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:13.424327 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.424300    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2035c390-696e-4873-bcb3-a6e13f656072-proxy-tls" pod "2035c390-696e-4873-bcb3-a6e13f656072" (UID: "2035c390-696e-4873-bcb3-a6e13f656072"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:13.437182 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.437158    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"]
Oct 30 22:23:13.437474 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.437433    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kserve-container" containerID="cri-o://c6a7a2408373a14dfb78838b01006312c37c059bc2143937c903ec146be4fb30" gracePeriod=30
Oct 30 22:23:13.437548 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.437471    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kube-rbac-proxy" containerID="cri-o://7b2ffe16515b5152aac651b51317d64770be5caa3aa613c6bc144d13ad8e2bba" gracePeriod=30
Oct 30 22:23:13.522186 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.522154    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jtfj\" (UniqueName: \"kubernetes.io/projected/2035c390-696e-4873-bcb3-a6e13f656072-kube-api-access-7jtfj\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:13.522186 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.522183    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2035c390-696e-4873-bcb3-a6e13f656072-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:13.522186 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.522192    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2035c390-696e-4873-bcb3-a6e13f656072-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:13.522421 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.522202    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2035c390-696e-4873-bcb3-a6e13f656072-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:13.522421 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.522211    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2035c390-696e-4873-bcb3-a6e13f656072-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:13.709615 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.709548    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-secondary-921a4c-predictor-659fd47bd5-trwfk_2035c390-696e-4873-bcb3-a6e13f656072/storage-initializer/1.log"
Oct 30 22:23:13.709751 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.709644    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk" event={"ID":"2035c390-696e-4873-bcb3-a6e13f656072","Type":"ContainerDied","Data":"7076bb1df0d9a243c27936368ac3e5f881415ac29373ec4546e694996544fc2b"}
Oct 30 22:23:13.709751 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.709655    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"
Oct 30 22:23:13.709751 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.709715    4947 scope.go:122] "RemoveContainer" containerID="d49e8dfdb6ea9ee9bc0906637ed58af760308146dc9ac2beddbde5aaecef7b01"
Oct 30 22:23:13.712137 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.712111    4947 generic.go:356] "Generic (PLEG): container finished" podID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerID="7b2ffe16515b5152aac651b51317d64770be5caa3aa613c6bc144d13ad8e2bba" exitCode=0
Oct 30 22:23:13.712222 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.712186    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" event={"ID":"6489abac-6c1b-4f5f-8b02-0a95fd735ee7","Type":"ContainerDied","Data":"7b2ffe16515b5152aac651b51317d64770be5caa3aa613c6bc144d13ad8e2bba"}
Oct 30 22:23:13.948264 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.948230    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"]
Oct 30 22:23:13.958066 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:13.958038    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-secondary-921a4c-predictor-659fd47bd5-trwfk"]
Oct 30 22:23:14.152579 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:14.152536    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Oct 30 22:23:14.152988 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:14.152962    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 30 22:23:14.157920 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:14.157896    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2035c390-696e-4873-bcb3-a6e13f656072" path="/var/lib/kubelet/pods/2035c390-696e-4873-bcb3-a6e13f656072/volumes"
Oct 30 22:23:18.185982 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.185957    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:23:18.364216 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.364184    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kube-api-access-jlj9h\" (UniqueName: \"kubernetes.io/projected/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kube-api-access-jlj9h\") pod \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") "
Oct 30 22:23:18.364411 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.364252    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-cabundle-cert\") pod \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") "
Oct 30 22:23:18.364411 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.364291    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-proxy-tls\" (UniqueName: \"kubernetes.io/secret/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-proxy-tls\") pod \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") "
Oct 30 22:23:18.364411 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.364396    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kserve-provision-location\") pod \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") "
Oct 30 22:23:18.364567 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.364427    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kube-rbac-proxy-sar-config\") pod \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\" (UID: \"6489abac-6c1b-4f5f-8b02-0a95fd735ee7\") "
Oct 30 22:23:18.364767 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.364738    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-cabundle-cert" pod "6489abac-6c1b-4f5f-8b02-0a95fd735ee7" (UID: "6489abac-6c1b-4f5f-8b02-0a95fd735ee7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:18.364899 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.364792    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kserve-provision-location" pod "6489abac-6c1b-4f5f-8b02-0a95fd735ee7" (UID: "6489abac-6c1b-4f5f-8b02-0a95fd735ee7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:18.364899 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.364826    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kube-rbac-proxy-sar-config" pod "6489abac-6c1b-4f5f-8b02-0a95fd735ee7" (UID: "6489abac-6c1b-4f5f-8b02-0a95fd735ee7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:18.366555 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.366526    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-proxy-tls" pod "6489abac-6c1b-4f5f-8b02-0a95fd735ee7" (UID: "6489abac-6c1b-4f5f-8b02-0a95fd735ee7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:18.366639 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.366586    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kube-api-access-jlj9h" pod "6489abac-6c1b-4f5f-8b02-0a95fd735ee7" (UID: "6489abac-6c1b-4f5f-8b02-0a95fd735ee7"). InnerVolumeSpecName "kube-api-access-jlj9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:18.465284 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.465248    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlj9h\" (UniqueName: \"kubernetes.io/projected/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kube-api-access-jlj9h\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:18.465284 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.465278    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:18.465491 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.465308    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:18.465491 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.465318    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:18.465491 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.465327    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6489abac-6c1b-4f5f-8b02-0a95fd735ee7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:18.736471 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.736394    4947 generic.go:356] "Generic (PLEG): container finished" podID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerID="c6a7a2408373a14dfb78838b01006312c37c059bc2143937c903ec146be4fb30" exitCode=0
Oct 30 22:23:18.736471 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.736450    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"
Oct 30 22:23:18.736657 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.736472    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" event={"ID":"6489abac-6c1b-4f5f-8b02-0a95fd735ee7","Type":"ContainerDied","Data":"c6a7a2408373a14dfb78838b01006312c37c059bc2143937c903ec146be4fb30"}
Oct 30 22:23:18.736657 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.736530    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68" event={"ID":"6489abac-6c1b-4f5f-8b02-0a95fd735ee7","Type":"ContainerDied","Data":"9dbd03175ae7bdde0f362a1d3cbbe2800b8ce689192b840a40c95c49714409d1"}
Oct 30 22:23:18.736657 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.736557    4947 scope.go:122] "RemoveContainer" containerID="7b2ffe16515b5152aac651b51317d64770be5caa3aa613c6bc144d13ad8e2bba"
Oct 30 22:23:18.744975 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.744961    4947 scope.go:122] "RemoveContainer" containerID="c6a7a2408373a14dfb78838b01006312c37c059bc2143937c903ec146be4fb30"
Oct 30 22:23:18.752065 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.752051    4947 scope.go:122] "RemoveContainer" containerID="fa3f0b19e7554e4e41bbbf44ae06eb29df87ccb9181a054ad6526023c10ff01f"
Oct 30 22:23:18.759117 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.759095    4947 scope.go:122] "RemoveContainer" containerID="7b2ffe16515b5152aac651b51317d64770be5caa3aa613c6bc144d13ad8e2bba"
Oct 30 22:23:18.759411 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:23:18.759390    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b2ffe16515b5152aac651b51317d64770be5caa3aa613c6bc144d13ad8e2bba\": container with ID starting with 7b2ffe16515b5152aac651b51317d64770be5caa3aa613c6bc144d13ad8e2bba not found: ID does not exist" containerID="7b2ffe16515b5152aac651b51317d64770be5caa3aa613c6bc144d13ad8e2bba"
Oct 30 22:23:18.759472 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.759424    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b2ffe16515b5152aac651b51317d64770be5caa3aa613c6bc144d13ad8e2bba"} err="failed to get container status \"7b2ffe16515b5152aac651b51317d64770be5caa3aa613c6bc144d13ad8e2bba\": rpc error: code = NotFound desc = could not find container \"7b2ffe16515b5152aac651b51317d64770be5caa3aa613c6bc144d13ad8e2bba\": container with ID starting with 7b2ffe16515b5152aac651b51317d64770be5caa3aa613c6bc144d13ad8e2bba not found: ID does not exist"
Oct 30 22:23:18.759472 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.759447    4947 scope.go:122] "RemoveContainer" containerID="c6a7a2408373a14dfb78838b01006312c37c059bc2143937c903ec146be4fb30"
Oct 30 22:23:18.759687 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:23:18.759667    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a7a2408373a14dfb78838b01006312c37c059bc2143937c903ec146be4fb30\": container with ID starting with c6a7a2408373a14dfb78838b01006312c37c059bc2143937c903ec146be4fb30 not found: ID does not exist" containerID="c6a7a2408373a14dfb78838b01006312c37c059bc2143937c903ec146be4fb30"
Oct 30 22:23:18.759755 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.759696    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a7a2408373a14dfb78838b01006312c37c059bc2143937c903ec146be4fb30"} err="failed to get container status \"c6a7a2408373a14dfb78838b01006312c37c059bc2143937c903ec146be4fb30\": rpc error: code = NotFound desc = could not find container \"c6a7a2408373a14dfb78838b01006312c37c059bc2143937c903ec146be4fb30\": container with ID starting with c6a7a2408373a14dfb78838b01006312c37c059bc2143937c903ec146be4fb30 not found: ID does not exist"
Oct 30 22:23:18.759755 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.759718    4947 scope.go:122] "RemoveContainer" containerID="fa3f0b19e7554e4e41bbbf44ae06eb29df87ccb9181a054ad6526023c10ff01f"
Oct 30 22:23:18.759990 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:23:18.759972    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3f0b19e7554e4e41bbbf44ae06eb29df87ccb9181a054ad6526023c10ff01f\": container with ID starting with fa3f0b19e7554e4e41bbbf44ae06eb29df87ccb9181a054ad6526023c10ff01f not found: ID does not exist" containerID="fa3f0b19e7554e4e41bbbf44ae06eb29df87ccb9181a054ad6526023c10ff01f"
Oct 30 22:23:18.760027 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.759996    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3f0b19e7554e4e41bbbf44ae06eb29df87ccb9181a054ad6526023c10ff01f"} err="failed to get container status \"fa3f0b19e7554e4e41bbbf44ae06eb29df87ccb9181a054ad6526023c10ff01f\": rpc error: code = NotFound desc = could not find container \"fa3f0b19e7554e4e41bbbf44ae06eb29df87ccb9181a054ad6526023c10ff01f\": container with ID starting with fa3f0b19e7554e4e41bbbf44ae06eb29df87ccb9181a054ad6526023c10ff01f not found: ID does not exist"
Oct 30 22:23:18.903948 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.903916    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"]
Oct 30 22:23:18.953902 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:18.953874    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-primary-921a4c-predictor-c9cc8c5d6-5pz68"]
Oct 30 22:23:20.157977 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:20.157949    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" path="/var/lib/kubelet/pods/6489abac-6c1b-4f5f-8b02-0a95fd735ee7/volumes"
Oct 30 22:23:21.506663 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:21.506622    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"]
Oct 30 22:23:21.507129 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:21.506957    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="agent" containerID="cri-o://021c35b3d9ae7294be1dceb6d5e1302a63de146ddbfa16a91318b29f22de8793" gracePeriod=30
Oct 30 22:23:21.507129 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:21.507017    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kube-rbac-proxy" containerID="cri-o://57f58cd1a836ead9342263a5d6b974637bf8c849522c50dee2c9c689e9bab97d" gracePeriod=30
Oct 30 22:23:21.507129 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:21.506958    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kserve-container" containerID="cri-o://cc339937956ab7eae85295365cfeb930cf28b22dbe73479821aa5380c293360f" gracePeriod=30
Oct 30 22:23:21.752723 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:21.752692    4947 generic.go:356] "Generic (PLEG): container finished" podID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerID="57f58cd1a836ead9342263a5d6b974637bf8c849522c50dee2c9c689e9bab97d" exitCode=0
Oct 30 22:23:21.752884 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:21.752769    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" event={"ID":"0cd5f470-2251-4f17-a5e5-8fbb823c5244","Type":"ContainerDied","Data":"57f58cd1a836ead9342263a5d6b974637bf8c849522c50dee2c9c689e9bab97d"}
Oct 30 22:23:22.160246 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.160217    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"]
Oct 30 22:23:22.160575 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.160561    4947 cpu_manager.go:414] "containerMap: removing container" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kserve-container"
Oct 30 22:23:22.160633 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.160577    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kserve-container"
Oct 30 22:23:22.160633 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.160597    4947 cpu_manager.go:414] "containerMap: removing container" podUID="2035c390-696e-4873-bcb3-a6e13f656072" containerName="storage-initializer"
Oct 30 22:23:22.160633 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.160601    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2035c390-696e-4873-bcb3-a6e13f656072" containerName="storage-initializer"
Oct 30 22:23:22.160633 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.160609    4947 cpu_manager.go:414] "containerMap: removing container" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kube-rbac-proxy"
Oct 30 22:23:22.160633 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.160614    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kube-rbac-proxy"
Oct 30 22:23:22.160633 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.160627    4947 cpu_manager.go:414] "containerMap: removing container" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="storage-initializer"
Oct 30 22:23:22.160633 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.160632    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="storage-initializer"
Oct 30 22:23:22.160836 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.160639    4947 cpu_manager.go:414] "containerMap: removing container" podUID="2035c390-696e-4873-bcb3-a6e13f656072" containerName="storage-initializer"
Oct 30 22:23:22.160836 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.160643    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2035c390-696e-4873-bcb3-a6e13f656072" containerName="storage-initializer"
Oct 30 22:23:22.160836 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.160698    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kserve-container"
Oct 30 22:23:22.160836 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.160707    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="2035c390-696e-4873-bcb3-a6e13f656072" containerName="storage-initializer"
Oct 30 22:23:22.160836 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.160714    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6489abac-6c1b-4f5f-8b02-0a95fd735ee7" containerName="kube-rbac-proxy"
Oct 30 22:23:22.160836 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.160721    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="2035c390-696e-4873-bcb3-a6e13f656072" containerName="storage-initializer"
Oct 30 22:23:22.163770 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.163754    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:22.218078 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.218047    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:23:22.218078 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.218047    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:23:22.218298 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.218095    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-init-fail-5bfa57-kube-rbac-proxy-sar-config\""
Oct 30 22:23:22.218298 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.218142    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"fail-s3-secret-5bfa57\""
Oct 30 22:23:22.218298 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.218165    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:23:22.220940 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.220923    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:23:22.229510 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.229479    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:23:22.266833 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.266813    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"fail-s3-sa-5bfa57-dockercfg-5l9lg\""
Oct 30 22:23:22.266833 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.266823    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-init-fail-5bfa57-predictor-serving-cert\""
Oct 30 22:23:22.268081 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.268063    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"]
Oct 30 22:23:22.296376 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.296327    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e683368-d2fc-473f-b168-b76269f76ffb-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") " pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:22.296500 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.296432    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgqsb\" (UniqueName: \"kubernetes.io/projected/8e683368-d2fc-473f-b168-b76269f76ffb-kube-api-access-mgqsb\") pod \"isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") " pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:22.296500 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.296466    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e683368-d2fc-473f-b168-b76269f76ffb-kserve-provision-location\") pod \"isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") " pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:22.296500 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.296491    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e683368-d2fc-473f-b168-b76269f76ffb-cabundle-cert\") pod \"isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") " pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:22.296597 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.296557    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e683368-d2fc-473f-b168-b76269f76ffb-proxy-tls\") pod \"isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") " pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:22.397535 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.397509    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e683368-d2fc-473f-b168-b76269f76ffb-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") " pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:22.397671 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.397548    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgqsb\" (UniqueName: \"kubernetes.io/projected/8e683368-d2fc-473f-b168-b76269f76ffb-kube-api-access-mgqsb\") pod \"isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") " pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:22.397671 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.397572    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e683368-d2fc-473f-b168-b76269f76ffb-kserve-provision-location\") pod \"isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") " pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:22.397671 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.397620    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e683368-d2fc-473f-b168-b76269f76ffb-cabundle-cert\") pod \"isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") " pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:22.397671 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.397666    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e683368-d2fc-473f-b168-b76269f76ffb-proxy-tls\") pod \"isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") " pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:22.398096 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.398073    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e683368-d2fc-473f-b168-b76269f76ffb-kserve-provision-location\") pod \"isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") " pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:22.398312 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.398295    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e683368-d2fc-473f-b168-b76269f76ffb-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") " pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:22.398390 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.398301    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e683368-d2fc-473f-b168-b76269f76ffb-cabundle-cert\") pod \"isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") " pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:22.400676 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.400654    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e683368-d2fc-473f-b168-b76269f76ffb-proxy-tls\") pod \"isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") " pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:22.460677 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.460617    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgqsb\" (UniqueName: \"kubernetes.io/projected/8e683368-d2fc-473f-b168-b76269f76ffb-kube-api-access-mgqsb\") pod \"isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") " pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:22.480072 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.480053    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:22.904862 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:22.904725    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"]
Oct 30 22:23:22.907325 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:23:22.907296    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e683368_d2fc_473f_b168_b76269f76ffb.slice/crio-f70ce7efd7e5544141b8101441afac582441a39397187cf5f3c87b924d39b2f6 WatchSource:0}: Error finding container f70ce7efd7e5544141b8101441afac582441a39397187cf5f3c87b924d39b2f6: Status 404 returned error can't find the container with id f70ce7efd7e5544141b8101441afac582441a39397187cf5f3c87b924d39b2f6
Oct 30 22:23:23.762739 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:23.762704    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6" event={"ID":"8e683368-d2fc-473f-b168-b76269f76ffb","Type":"ContainerStarted","Data":"756cd372bc3d4771d6cb682e048bd4ad1440ab3e023655756fb70845f61cae4a"}
Oct 30 22:23:23.762916 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:23.762768    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6" event={"ID":"8e683368-d2fc-473f-b168-b76269f76ffb","Type":"ContainerStarted","Data":"f70ce7efd7e5544141b8101441afac582441a39397187cf5f3c87b924d39b2f6"}
Oct 30 22:23:24.374356 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:24.374292    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Oct 30 22:23:26.777256 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:26.777164    4947 generic.go:356] "Generic (PLEG): container finished" podID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerID="cc339937956ab7eae85295365cfeb930cf28b22dbe73479821aa5380c293360f" exitCode=0
Oct 30 22:23:26.777648 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:26.777244    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" event={"ID":"0cd5f470-2251-4f17-a5e5-8fbb823c5244","Type":"ContainerDied","Data":"cc339937956ab7eae85295365cfeb930cf28b22dbe73479821aa5380c293360f"}
Oct 30 22:23:26.778754 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:26.778736    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6_8e683368-d2fc-473f-b168-b76269f76ffb/storage-initializer/0.log"
Oct 30 22:23:26.778824 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:26.778768    4947 generic.go:356] "Generic (PLEG): container finished" podID="8e683368-d2fc-473f-b168-b76269f76ffb" containerID="756cd372bc3d4771d6cb682e048bd4ad1440ab3e023655756fb70845f61cae4a" exitCode=1
Oct 30 22:23:26.778867 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:26.778821    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6" event={"ID":"8e683368-d2fc-473f-b168-b76269f76ffb","Type":"ContainerDied","Data":"756cd372bc3d4771d6cb682e048bd4ad1440ab3e023655756fb70845f61cae4a"}
Oct 30 22:23:27.785264 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:27.785232    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6_8e683368-d2fc-473f-b168-b76269f76ffb/storage-initializer/0.log"
Oct 30 22:23:27.785778 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:27.785317    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6" event={"ID":"8e683368-d2fc-473f-b168-b76269f76ffb","Type":"ContainerStarted","Data":"463c8b8244e77d08059add04ce3e2838a3f22f1bd4230f8758f1f28bd37fb234"}
Oct 30 22:23:29.048488 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:23:29.048449    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:23:29.374222 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:29.374133    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Oct 30 22:23:29.378594 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:29.378561    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:5000: connect: connection refused"
Oct 30 22:23:29.378884 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:29.378862    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:23:30.800112 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:30.800084    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6_8e683368-d2fc-473f-b168-b76269f76ffb/storage-initializer/1.log"
Oct 30 22:23:30.800578 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:30.800510    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6_8e683368-d2fc-473f-b168-b76269f76ffb/storage-initializer/0.log"
Oct 30 22:23:30.800578 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:30.800540    4947 generic.go:356] "Generic (PLEG): container finished" podID="8e683368-d2fc-473f-b168-b76269f76ffb" containerID="463c8b8244e77d08059add04ce3e2838a3f22f1bd4230f8758f1f28bd37fb234" exitCode=1
Oct 30 22:23:30.800653 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:30.800616    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6" event={"ID":"8e683368-d2fc-473f-b168-b76269f76ffb","Type":"ContainerDied","Data":"463c8b8244e77d08059add04ce3e2838a3f22f1bd4230f8758f1f28bd37fb234"}
Oct 30 22:23:30.800697 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:30.800685    4947 scope.go:122] "RemoveContainer" containerID="756cd372bc3d4771d6cb682e048bd4ad1440ab3e023655756fb70845f61cae4a"
Oct 30 22:23:30.801000 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:30.800983    4947 scope.go:122] "RemoveContainer" containerID="756cd372bc3d4771d6cb682e048bd4ad1440ab3e023655756fb70845f61cae4a"
Oct 30 22:23:30.811636 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:23:30.811615    4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6_e2e-gw1-bc5112d610(8e683368-d2fc-473f-b168-b76269f76ffb)\"" pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6" podUID="8e683368-d2fc-473f-b168-b76269f76ffb"
Oct 30 22:23:30.811843 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:23:30.811820    4947 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6_e2e-gw1-bc5112d610_8e683368-d2fc-473f-b168-b76269f76ffb_0 in pod sandbox f70ce7efd7e5544141b8101441afac582441a39397187cf5f3c87b924d39b2f6 from index: no such id: '756cd372bc3d4771d6cb682e048bd4ad1440ab3e023655756fb70845f61cae4a'" containerID="756cd372bc3d4771d6cb682e048bd4ad1440ab3e023655756fb70845f61cae4a"
Oct 30 22:23:30.811888 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:30.811852    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756cd372bc3d4771d6cb682e048bd4ad1440ab3e023655756fb70845f61cae4a"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6_e2e-gw1-bc5112d610_8e683368-d2fc-473f-b168-b76269f76ffb_0 in pod sandbox f70ce7efd7e5544141b8101441afac582441a39397187cf5f3c87b924d39b2f6 from index: no such id: '756cd372bc3d4771d6cb682e048bd4ad1440ab3e023655756fb70845f61cae4a'"
Oct 30 22:23:31.746948 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:31.746914    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"]
Oct 30 22:23:31.806162 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:31.806135    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6_8e683368-d2fc-473f-b168-b76269f76ffb/storage-initializer/1.log"
Oct 30 22:23:31.934459 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:31.934433    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6_8e683368-d2fc-473f-b168-b76269f76ffb/storage-initializer/1.log"
Oct 30 22:23:31.938072 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:31.938053    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:31.979199 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:31.979165    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e683368-d2fc-473f-b168-b76269f76ffb-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e683368-d2fc-473f-b168-b76269f76ffb-cabundle-cert\") pod \"8e683368-d2fc-473f-b168-b76269f76ffb\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") "
Oct 30 22:23:31.979412 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:31.979205    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e683368-d2fc-473f-b168-b76269f76ffb-kube-api-access-mgqsb\" (UniqueName: \"kubernetes.io/projected/8e683368-d2fc-473f-b168-b76269f76ffb-kube-api-access-mgqsb\") pod \"8e683368-d2fc-473f-b168-b76269f76ffb\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") "
Oct 30 22:23:31.979412 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:31.979239    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e683368-d2fc-473f-b168-b76269f76ffb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e683368-d2fc-473f-b168-b76269f76ffb-kube-rbac-proxy-sar-config\") pod \"8e683368-d2fc-473f-b168-b76269f76ffb\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") "
Oct 30 22:23:31.979412 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:31.979265    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e683368-d2fc-473f-b168-b76269f76ffb-proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e683368-d2fc-473f-b168-b76269f76ffb-proxy-tls\") pod \"8e683368-d2fc-473f-b168-b76269f76ffb\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") "
Oct 30 22:23:31.979412 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:31.979307    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e683368-d2fc-473f-b168-b76269f76ffb-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e683368-d2fc-473f-b168-b76269f76ffb-kserve-provision-location\") pod \"8e683368-d2fc-473f-b168-b76269f76ffb\" (UID: \"8e683368-d2fc-473f-b168-b76269f76ffb\") "
Oct 30 22:23:31.979721 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:31.979691    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e683368-d2fc-473f-b168-b76269f76ffb-cabundle-cert" pod "8e683368-d2fc-473f-b168-b76269f76ffb" (UID: "8e683368-d2fc-473f-b168-b76269f76ffb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:31.979721 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:31.979699    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e683368-d2fc-473f-b168-b76269f76ffb-kserve-provision-location" pod "8e683368-d2fc-473f-b168-b76269f76ffb" (UID: "8e683368-d2fc-473f-b168-b76269f76ffb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:31.979854 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:31.979736    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e683368-d2fc-473f-b168-b76269f76ffb-kube-rbac-proxy-sar-config" pod "8e683368-d2fc-473f-b168-b76269f76ffb" (UID: "8e683368-d2fc-473f-b168-b76269f76ffb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:31.981538 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:31.981517    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e683368-d2fc-473f-b168-b76269f76ffb-proxy-tls" pod "8e683368-d2fc-473f-b168-b76269f76ffb" (UID: "8e683368-d2fc-473f-b168-b76269f76ffb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:31.981661 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:31.981643    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e683368-d2fc-473f-b168-b76269f76ffb-kube-api-access-mgqsb" pod "8e683368-d2fc-473f-b168-b76269f76ffb" (UID: "8e683368-d2fc-473f-b168-b76269f76ffb"). InnerVolumeSpecName "kube-api-access-mgqsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:32.080544 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:32.080510    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e683368-d2fc-473f-b168-b76269f76ffb-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:32.080544 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:32.080539    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgqsb\" (UniqueName: \"kubernetes.io/projected/8e683368-d2fc-473f-b168-b76269f76ffb-kube-api-access-mgqsb\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:32.080544 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:32.080549    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e683368-d2fc-473f-b168-b76269f76ffb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:32.080772 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:32.080559    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e683368-d2fc-473f-b168-b76269f76ffb-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:32.080772 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:32.080567    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e683368-d2fc-473f-b168-b76269f76ffb-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:32.812028 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:32.812001    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6_8e683368-d2fc-473f-b168-b76269f76ffb/storage-initializer/1.log"
Oct 30 22:23:32.812524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:32.812099    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"
Oct 30 22:23:32.812524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:32.812101    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6" event={"ID":"8e683368-d2fc-473f-b168-b76269f76ffb","Type":"ContainerDied","Data":"f70ce7efd7e5544141b8101441afac582441a39397187cf5f3c87b924d39b2f6"}
Oct 30 22:23:32.812524 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:32.812181    4947 scope.go:122] "RemoveContainer" containerID="463c8b8244e77d08059add04ce3e2838a3f22f1bd4230f8758f1f28bd37fb234"
Oct 30 22:23:33.096041 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:33.095956    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"]
Oct 30 22:23:33.101472 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:33.101439    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-init-fail-5bfa57-predictor-7cd87559cf-8lwl6"]
Oct 30 22:23:34.158504 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:34.158470    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e683368-d2fc-473f-b168-b76269f76ffb" path="/var/lib/kubelet/pods/8e683368-d2fc-473f-b168-b76269f76ffb/volumes"
Oct 30 22:23:34.374543 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:34.374506    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Oct 30 22:23:34.374704 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:34.374636    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:23:39.374172 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:39.374128    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Oct 30 22:23:39.378496 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:39.378469    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:5000: connect: connection refused"
Oct 30 22:23:39.378908 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:39.378881    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:23:44.374575 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:44.374527    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Oct 30 22:23:49.374210 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:49.374162    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Oct 30 22:23:49.378514 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:49.378485    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:5000: connect: connection refused"
Oct 30 22:23:49.378658 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:49.378631    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:23:49.378855 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:49.378835    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:23:49.378934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:49.378923    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:23:51.664665 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.664645    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:23:51.740983 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.740943    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kserve-provision-location\") pod \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") "
Oct 30 22:23:51.741162 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.740989    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kube-rbac-proxy-sar-config\") pod \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") "
Oct 30 22:23:51.741162 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.741040    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kube-api-access-6g5bn\" (UniqueName: \"kubernetes.io/projected/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kube-api-access-6g5bn\") pod \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") "
Oct 30 22:23:51.741162 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.741092    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd5f470-2251-4f17-a5e5-8fbb823c5244-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cd5f470-2251-4f17-a5e5-8fbb823c5244-cabundle-cert\") pod \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") "
Oct 30 22:23:51.741162 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.741133    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cd5f470-2251-4f17-a5e5-8fbb823c5244-proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cd5f470-2251-4f17-a5e5-8fbb823c5244-proxy-tls\") pod \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\" (UID: \"0cd5f470-2251-4f17-a5e5-8fbb823c5244\") "
Oct 30 22:23:51.741421 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.741277    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kserve-provision-location" pod "0cd5f470-2251-4f17-a5e5-8fbb823c5244" (UID: "0cd5f470-2251-4f17-a5e5-8fbb823c5244"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:51.741475 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.741444    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kube-rbac-proxy-sar-config" pod "0cd5f470-2251-4f17-a5e5-8fbb823c5244" (UID: "0cd5f470-2251-4f17-a5e5-8fbb823c5244"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:51.741518 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.741491    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd5f470-2251-4f17-a5e5-8fbb823c5244-cabundle-cert" pod "0cd5f470-2251-4f17-a5e5-8fbb823c5244" (UID: "0cd5f470-2251-4f17-a5e5-8fbb823c5244"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:51.743413 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.743388    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kube-api-access-6g5bn" pod "0cd5f470-2251-4f17-a5e5-8fbb823c5244" (UID: "0cd5f470-2251-4f17-a5e5-8fbb823c5244"). InnerVolumeSpecName "kube-api-access-6g5bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:51.743499 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.743478    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd5f470-2251-4f17-a5e5-8fbb823c5244-proxy-tls" pod "0cd5f470-2251-4f17-a5e5-8fbb823c5244" (UID: "0cd5f470-2251-4f17-a5e5-8fbb823c5244"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:51.842092 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.842059    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:51.842092 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.842087    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g5bn\" (UniqueName: \"kubernetes.io/projected/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kube-api-access-6g5bn\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:51.842092 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.842098    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cd5f470-2251-4f17-a5e5-8fbb823c5244-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:51.842328 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.842107    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cd5f470-2251-4f17-a5e5-8fbb823c5244-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:51.842328 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.842125    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cd5f470-2251-4f17-a5e5-8fbb823c5244-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:51.893757 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.893721    4947 generic.go:356] "Generic (PLEG): container finished" podID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerID="021c35b3d9ae7294be1dceb6d5e1302a63de146ddbfa16a91318b29f22de8793" exitCode=0
Oct 30 22:23:51.893932 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.893794    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"
Oct 30 22:23:51.893932 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.893807    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" event={"ID":"0cd5f470-2251-4f17-a5e5-8fbb823c5244","Type":"ContainerDied","Data":"021c35b3d9ae7294be1dceb6d5e1302a63de146ddbfa16a91318b29f22de8793"}
Oct 30 22:23:51.893932 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.893886    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh" event={"ID":"0cd5f470-2251-4f17-a5e5-8fbb823c5244","Type":"ContainerDied","Data":"33d6c6a7d3d19c90a0939d292d051b1500645de1e0eb740c9543ea95b06326a6"}
Oct 30 22:23:51.893932 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.893929    4947 scope.go:122] "RemoveContainer" containerID="021c35b3d9ae7294be1dceb6d5e1302a63de146ddbfa16a91318b29f22de8793"
Oct 30 22:23:51.902548 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.902530    4947 scope.go:122] "RemoveContainer" containerID="57f58cd1a836ead9342263a5d6b974637bf8c849522c50dee2c9c689e9bab97d"
Oct 30 22:23:51.911398 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.911384    4947 scope.go:122] "RemoveContainer" containerID="cc339937956ab7eae85295365cfeb930cf28b22dbe73479821aa5380c293360f"
Oct 30 22:23:51.919066 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.919051    4947 scope.go:122] "RemoveContainer" containerID="c164f1346637c953b11ff3b3e3bfa735a11b83ab6776a84aecac142241dcda5c"
Oct 30 22:23:51.926183 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.926165    4947 scope.go:122] "RemoveContainer" containerID="021c35b3d9ae7294be1dceb6d5e1302a63de146ddbfa16a91318b29f22de8793"
Oct 30 22:23:51.926442 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:23:51.926422    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"021c35b3d9ae7294be1dceb6d5e1302a63de146ddbfa16a91318b29f22de8793\": container with ID starting with 021c35b3d9ae7294be1dceb6d5e1302a63de146ddbfa16a91318b29f22de8793 not found: ID does not exist" containerID="021c35b3d9ae7294be1dceb6d5e1302a63de146ddbfa16a91318b29f22de8793"
Oct 30 22:23:51.926498 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.926452    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021c35b3d9ae7294be1dceb6d5e1302a63de146ddbfa16a91318b29f22de8793"} err="failed to get container status \"021c35b3d9ae7294be1dceb6d5e1302a63de146ddbfa16a91318b29f22de8793\": rpc error: code = NotFound desc = could not find container \"021c35b3d9ae7294be1dceb6d5e1302a63de146ddbfa16a91318b29f22de8793\": container with ID starting with 021c35b3d9ae7294be1dceb6d5e1302a63de146ddbfa16a91318b29f22de8793 not found: ID does not exist"
Oct 30 22:23:51.926498 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.926469    4947 scope.go:122] "RemoveContainer" containerID="57f58cd1a836ead9342263a5d6b974637bf8c849522c50dee2c9c689e9bab97d"
Oct 30 22:23:51.926746 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:23:51.926731    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f58cd1a836ead9342263a5d6b974637bf8c849522c50dee2c9c689e9bab97d\": container with ID starting with 57f58cd1a836ead9342263a5d6b974637bf8c849522c50dee2c9c689e9bab97d not found: ID does not exist" containerID="57f58cd1a836ead9342263a5d6b974637bf8c849522c50dee2c9c689e9bab97d"
Oct 30 22:23:51.926795 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.926749    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f58cd1a836ead9342263a5d6b974637bf8c849522c50dee2c9c689e9bab97d"} err="failed to get container status \"57f58cd1a836ead9342263a5d6b974637bf8c849522c50dee2c9c689e9bab97d\": rpc error: code = NotFound desc = could not find container \"57f58cd1a836ead9342263a5d6b974637bf8c849522c50dee2c9c689e9bab97d\": container with ID starting with 57f58cd1a836ead9342263a5d6b974637bf8c849522c50dee2c9c689e9bab97d not found: ID does not exist"
Oct 30 22:23:51.926795 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.926769    4947 scope.go:122] "RemoveContainer" containerID="cc339937956ab7eae85295365cfeb930cf28b22dbe73479821aa5380c293360f"
Oct 30 22:23:51.927017 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:23:51.927000    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc339937956ab7eae85295365cfeb930cf28b22dbe73479821aa5380c293360f\": container with ID starting with cc339937956ab7eae85295365cfeb930cf28b22dbe73479821aa5380c293360f not found: ID does not exist" containerID="cc339937956ab7eae85295365cfeb930cf28b22dbe73479821aa5380c293360f"
Oct 30 22:23:51.927056 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.927032    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc339937956ab7eae85295365cfeb930cf28b22dbe73479821aa5380c293360f"} err="failed to get container status \"cc339937956ab7eae85295365cfeb930cf28b22dbe73479821aa5380c293360f\": rpc error: code = NotFound desc = could not find container \"cc339937956ab7eae85295365cfeb930cf28b22dbe73479821aa5380c293360f\": container with ID starting with cc339937956ab7eae85295365cfeb930cf28b22dbe73479821aa5380c293360f not found: ID does not exist"
Oct 30 22:23:51.927056 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.927048    4947 scope.go:122] "RemoveContainer" containerID="c164f1346637c953b11ff3b3e3bfa735a11b83ab6776a84aecac142241dcda5c"
Oct 30 22:23:51.927267 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:23:51.927250    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c164f1346637c953b11ff3b3e3bfa735a11b83ab6776a84aecac142241dcda5c\": container with ID starting with c164f1346637c953b11ff3b3e3bfa735a11b83ab6776a84aecac142241dcda5c not found: ID does not exist" containerID="c164f1346637c953b11ff3b3e3bfa735a11b83ab6776a84aecac142241dcda5c"
Oct 30 22:23:51.927309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:51.927272    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c164f1346637c953b11ff3b3e3bfa735a11b83ab6776a84aecac142241dcda5c"} err="failed to get container status \"c164f1346637c953b11ff3b3e3bfa735a11b83ab6776a84aecac142241dcda5c\": rpc error: code = NotFound desc = could not find container \"c164f1346637c953b11ff3b3e3bfa735a11b83ab6776a84aecac142241dcda5c\": container with ID starting with c164f1346637c953b11ff3b3e3bfa735a11b83ab6776a84aecac142241dcda5c not found: ID does not exist"
Oct 30 22:23:52.072241 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:52.072203    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"]
Oct 30 22:23:52.075699 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:52.075674    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-sklearn-batcher-custom-predictor-598c78fb67-mhqbh"]
Oct 30 22:23:52.158803 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:23:52.158730    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" path="/var/lib/kubelet/pods/0cd5f470-2251-4f17-a5e5-8fbb823c5244/volumes"
Oct 30 22:23:59.048830 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:23:59.048794    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:24:29.048109 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:24:29.048073    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:24:59.047944 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:24:59.047910    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:25:25.985538 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985501    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"]
Oct 30 22:25:25.985934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985815    4947 cpu_manager.go:414] "containerMap: removing container" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kube-rbac-proxy"
Oct 30 22:25:25.985934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985826    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kube-rbac-proxy"
Oct 30 22:25:25.985934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985842    4947 cpu_manager.go:414] "containerMap: removing container" podUID="8e683368-d2fc-473f-b168-b76269f76ffb" containerName="storage-initializer"
Oct 30 22:25:25.985934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985847    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e683368-d2fc-473f-b168-b76269f76ffb" containerName="storage-initializer"
Oct 30 22:25:25.985934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985860    4947 cpu_manager.go:414] "containerMap: removing container" podUID="8e683368-d2fc-473f-b168-b76269f76ffb" containerName="storage-initializer"
Oct 30 22:25:25.985934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985866    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e683368-d2fc-473f-b168-b76269f76ffb" containerName="storage-initializer"
Oct 30 22:25:25.985934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985872    4947 cpu_manager.go:414] "containerMap: removing container" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="storage-initializer"
Oct 30 22:25:25.985934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985877    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="storage-initializer"
Oct 30 22:25:25.985934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985886    4947 cpu_manager.go:414] "containerMap: removing container" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kserve-container"
Oct 30 22:25:25.985934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985891    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kserve-container"
Oct 30 22:25:25.985934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985899    4947 cpu_manager.go:414] "containerMap: removing container" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="agent"
Oct 30 22:25:25.985934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985903    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="agent"
Oct 30 22:25:25.986273 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985950    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kube-rbac-proxy"
Oct 30 22:25:25.986273 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985959    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e683368-d2fc-473f-b168-b76269f76ffb" containerName="storage-initializer"
Oct 30 22:25:25.986273 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985966    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e683368-d2fc-473f-b168-b76269f76ffb" containerName="storage-initializer"
Oct 30 22:25:25.986273 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985973    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="kserve-container"
Oct 30 22:25:25.986273 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.985978    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd5f470-2251-4f17-a5e5-8fbb823c5244" containerName="agent"
Oct 30 22:25:25.989344 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:25.989311    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:26.052960 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.052930    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:26.052960 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.052961    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/767f7ae4-f581-4bbe-a746-2cd359ebeff5-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:26.053136 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.053046    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/767f7ae4-f581-4bbe-a746-2cd359ebeff5-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:26.053136 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.053083    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kqp5\" (UniqueName: \"kubernetes.io/projected/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kube-api-access-8kqp5\") pod \"isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:26.053205 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.053141    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:26.094078 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.094038    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:25:26.094078 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.094038    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:25:26.094078 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.094076    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:25:26.094363 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.094038    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:25:26.094363 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.094043    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Oct 30 22:25:26.094363 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.094043    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:25:26.094363 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.094038    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Oct 30 22:25:26.095622 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.095605    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"]
Oct 30 22:25:26.153570 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.153537    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:26.153570 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.153575    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:26.153823 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.153596    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/767f7ae4-f581-4bbe-a746-2cd359ebeff5-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:26.153823 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.153635    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/767f7ae4-f581-4bbe-a746-2cd359ebeff5-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:26.153823 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.153692    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kqp5\" (UniqueName: \"kubernetes.io/projected/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kube-api-access-8kqp5\") pod \"isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:26.154113 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.154083    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:26.154281 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.154259    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:26.154403 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.154381    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/767f7ae4-f581-4bbe-a746-2cd359ebeff5-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:26.156237 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.156217    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/767f7ae4-f581-4bbe-a746-2cd359ebeff5-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:26.218655 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.218618    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kqp5\" (UniqueName: \"kubernetes.io/projected/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kube-api-access-8kqp5\") pod \"isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:26.305764 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.305730    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:26.731469 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:26.731444    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"]
Oct 30 22:25:26.734194 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:25:26.734164    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767f7ae4_f581_4bbe_a746_2cd359ebeff5.slice/crio-dcfb9921bfb5260043c4e0f167dc01c6b5cf762a0cc638dcb7b4c14e74133f21 WatchSource:0}: Error finding container dcfb9921bfb5260043c4e0f167dc01c6b5cf762a0cc638dcb7b4c14e74133f21: Status 404 returned error can't find the container with id dcfb9921bfb5260043c4e0f167dc01c6b5cf762a0cc638dcb7b4c14e74133f21
Oct 30 22:25:27.266847 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:27.266814    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" event={"ID":"767f7ae4-f581-4bbe-a746-2cd359ebeff5","Type":"ContainerStarted","Data":"ba36d83858228efb7df0c7617dea6b1a3391af0a7a489109d78414554e6b93f0"}
Oct 30 22:25:27.267308 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:27.266881    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" event={"ID":"767f7ae4-f581-4bbe-a746-2cd359ebeff5","Type":"ContainerStarted","Data":"dcfb9921bfb5260043c4e0f167dc01c6b5cf762a0cc638dcb7b4c14e74133f21"}
Oct 30 22:25:29.047914 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:25:29.047882    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:25:31.285804 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:31.285771    4947 generic.go:356] "Generic (PLEG): container finished" podID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerID="ba36d83858228efb7df0c7617dea6b1a3391af0a7a489109d78414554e6b93f0" exitCode=0
Oct 30 22:25:31.285804 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:31.285805    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" event={"ID":"767f7ae4-f581-4bbe-a746-2cd359ebeff5","Type":"ContainerDied","Data":"ba36d83858228efb7df0c7617dea6b1a3391af0a7a489109d78414554e6b93f0"}
Oct 30 22:25:31.287322 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:31.287307    4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:25:55.414402 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:55.414367    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" event={"ID":"767f7ae4-f581-4bbe-a746-2cd359ebeff5","Type":"ContainerStarted","Data":"cd873534ead558a9af1650d74d4ce5562008d84f3f08f7ac16c2721a5c3d46c8"}
Oct 30 22:25:55.414843 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:55.414435    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" event={"ID":"767f7ae4-f581-4bbe-a746-2cd359ebeff5","Type":"ContainerStarted","Data":"97f081b630d70ff9210e46ce56423bc8bf42fd50739b1ffae6589a6e2884b6bf"}
Oct 30 22:25:55.414843 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:55.414639    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:55.414843 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:55.414698    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:25:55.416089 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:55.416064    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 22:25:55.539488 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:55.539439    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" podStartSLOduration=3.030172633 podStartE2EDuration="30.539424962s" totalImagesPullingTime="23.652393423s" totalInitContainerRuntime="3.856858906s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:55.538402493 +0000 UTC m=+1053.537750056" watchObservedRunningTime="2026-10-30 22:25:55.539424962 +0000 UTC m=+1053.538772524"
Oct 30 22:25:56.418419 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:25:56.418385    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 22:25:59.048698 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:25:59.048664    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:26:01.422670 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:01.422641    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:26:01.423160 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:01.423096    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 22:26:11.423158 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:11.423118    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 22:26:21.423074 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:21.423035    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 22:26:27.608983 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:27.608948    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-5mdb7-fg8fs"]
Oct 30 22:26:27.975588 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:27.975498    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5mdb7-fg8fs"]
Oct 30 22:26:27.975743 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:27.975598    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5mdb7-fg8fs"
Oct 30 22:26:28.054284 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:28.054252    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79347464-bcf5-4fc4-95d7-6e6c406d0332-trusted-ca-bundle\") pod \"periodic-gathering-5mdb7-fg8fs\" (UID: \"79347464-bcf5-4fc4-95d7-6e6c406d0332\") " pod="openshift-insights/periodic-gathering-5mdb7-fg8fs"
Oct 30 22:26:28.054493 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:28.054297    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/79347464-bcf5-4fc4-95d7-6e6c406d0332-archives-path\") pod \"periodic-gathering-5mdb7-fg8fs\" (UID: \"79347464-bcf5-4fc4-95d7-6e6c406d0332\") " pod="openshift-insights/periodic-gathering-5mdb7-fg8fs"
Oct 30 22:26:28.054493 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:28.054395    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79347464-bcf5-4fc4-95d7-6e6c406d0332-service-ca-bundle\") pod \"periodic-gathering-5mdb7-fg8fs\" (UID: \"79347464-bcf5-4fc4-95d7-6e6c406d0332\") " pod="openshift-insights/periodic-gathering-5mdb7-fg8fs"
Oct 30 22:26:28.054576 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:28.054502    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28h64\" (UniqueName: \"kubernetes.io/projected/79347464-bcf5-4fc4-95d7-6e6c406d0332-kube-api-access-28h64\") pod \"periodic-gathering-5mdb7-fg8fs\" (UID: \"79347464-bcf5-4fc4-95d7-6e6c406d0332\") " pod="openshift-insights/periodic-gathering-5mdb7-fg8fs"
Oct 30 22:26:28.155244 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:28.155212    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28h64\" (UniqueName: \"kubernetes.io/projected/79347464-bcf5-4fc4-95d7-6e6c406d0332-kube-api-access-28h64\") pod \"periodic-gathering-5mdb7-fg8fs\" (UID: \"79347464-bcf5-4fc4-95d7-6e6c406d0332\") " pod="openshift-insights/periodic-gathering-5mdb7-fg8fs"
Oct 30 22:26:28.155244 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:28.155255    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79347464-bcf5-4fc4-95d7-6e6c406d0332-trusted-ca-bundle\") pod \"periodic-gathering-5mdb7-fg8fs\" (UID: \"79347464-bcf5-4fc4-95d7-6e6c406d0332\") " pod="openshift-insights/periodic-gathering-5mdb7-fg8fs"
Oct 30 22:26:28.155525 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:28.155296    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/79347464-bcf5-4fc4-95d7-6e6c406d0332-archives-path\") pod \"periodic-gathering-5mdb7-fg8fs\" (UID: \"79347464-bcf5-4fc4-95d7-6e6c406d0332\") " pod="openshift-insights/periodic-gathering-5mdb7-fg8fs"
Oct 30 22:26:28.155525 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:28.155324    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79347464-bcf5-4fc4-95d7-6e6c406d0332-service-ca-bundle\") pod \"periodic-gathering-5mdb7-fg8fs\" (UID: \"79347464-bcf5-4fc4-95d7-6e6c406d0332\") " pod="openshift-insights/periodic-gathering-5mdb7-fg8fs"
Oct 30 22:26:28.155750 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:28.155727    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/79347464-bcf5-4fc4-95d7-6e6c406d0332-archives-path\") pod \"periodic-gathering-5mdb7-fg8fs\" (UID: \"79347464-bcf5-4fc4-95d7-6e6c406d0332\") " pod="openshift-insights/periodic-gathering-5mdb7-fg8fs"
Oct 30 22:26:28.156043 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:28.156023    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79347464-bcf5-4fc4-95d7-6e6c406d0332-service-ca-bundle\") pod \"periodic-gathering-5mdb7-fg8fs\" (UID: \"79347464-bcf5-4fc4-95d7-6e6c406d0332\") " pod="openshift-insights/periodic-gathering-5mdb7-fg8fs"
Oct 30 22:26:28.156504 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:28.156484    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79347464-bcf5-4fc4-95d7-6e6c406d0332-trusted-ca-bundle\") pod \"periodic-gathering-5mdb7-fg8fs\" (UID: \"79347464-bcf5-4fc4-95d7-6e6c406d0332\") " pod="openshift-insights/periodic-gathering-5mdb7-fg8fs"
Oct 30 22:26:28.218122 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:28.218086    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28h64\" (UniqueName: \"kubernetes.io/projected/79347464-bcf5-4fc4-95d7-6e6c406d0332-kube-api-access-28h64\") pod \"periodic-gathering-5mdb7-fg8fs\" (UID: \"79347464-bcf5-4fc4-95d7-6e6c406d0332\") " pod="openshift-insights/periodic-gathering-5mdb7-fg8fs"
Oct 30 22:26:28.313759 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:28.313707    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5mdb7-fg8fs"
Oct 30 22:26:28.687420 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:28.687326    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5mdb7-fg8fs"]
Oct 30 22:26:28.690223 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:26:28.690196    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79347464_bcf5_4fc4_95d7_6e6c406d0332.slice/crio-76af867ec18d33806a629679215cb0580de224e2b2f17a58cd0d1b4dbcd68531 WatchSource:0}: Error finding container 76af867ec18d33806a629679215cb0580de224e2b2f17a58cd0d1b4dbcd68531: Status 404 returned error can't find the container with id 76af867ec18d33806a629679215cb0580de224e2b2f17a58cd0d1b4dbcd68531
Oct 30 22:26:29.047611 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:26:29.047578    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:26:29.550546 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:29.550515    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5mdb7-fg8fs" event={"ID":"79347464-bcf5-4fc4-95d7-6e6c406d0332","Type":"ContainerStarted","Data":"0994c2e944b2d5a9c3d7fec8c98faf87db54c3d15b77f79c4cce6ea9c7958f84"}
Oct 30 22:26:29.550733 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:29.550570    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5mdb7-fg8fs" event={"ID":"79347464-bcf5-4fc4-95d7-6e6c406d0332","Type":"ContainerStarted","Data":"76af867ec18d33806a629679215cb0580de224e2b2f17a58cd0d1b4dbcd68531"}
Oct 30 22:26:29.719232 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:29.719182    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-5mdb7-fg8fs" podStartSLOduration=2.719169777 podStartE2EDuration="2.719169777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:29.718501671 +0000 UTC m=+1087.717849234" watchObservedRunningTime="2026-10-30 22:26:29.719169777 +0000 UTC m=+1087.718517341"
Oct 30 22:26:31.423285 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:31.423239    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 22:26:41.423885 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:41.423840    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 22:26:51.423137 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:26:51.423092    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 22:26:59.048961 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:26:59.048919    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:27:01.424263 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:01.424235    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:27:07.147793 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:07.147716    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"]
Oct 30 22:27:07.148184 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:07.147952    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kserve-container" containerID="cri-o://97f081b630d70ff9210e46ce56423bc8bf42fd50739b1ffae6589a6e2884b6bf" gracePeriod=30
Oct 30 22:27:07.148184 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:07.147986    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kube-rbac-proxy" containerID="cri-o://cd873534ead558a9af1650d74d4ce5562008d84f3f08f7ac16c2721a5c3d46c8" gracePeriod=30
Oct 30 22:27:07.702185 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:07.702147    4947 generic.go:356] "Generic (PLEG): container finished" podID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerID="cd873534ead558a9af1650d74d4ce5562008d84f3f08f7ac16c2721a5c3d46c8" exitCode=0
Oct 30 22:27:07.702185 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:07.702168    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" event={"ID":"767f7ae4-f581-4bbe-a746-2cd359ebeff5","Type":"ContainerDied","Data":"cd873534ead558a9af1650d74d4ce5562008d84f3f08f7ac16c2721a5c3d46c8"}
Oct 30 22:27:11.418642 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:11.418598    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Oct 30 22:27:11.423893 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:11.423865    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 22:27:11.721278 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:11.721244    4947 generic.go:356] "Generic (PLEG): container finished" podID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerID="97f081b630d70ff9210e46ce56423bc8bf42fd50739b1ffae6589a6e2884b6bf" exitCode=0
Oct 30 22:27:11.721433 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:11.721276    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" event={"ID":"767f7ae4-f581-4bbe-a746-2cd359ebeff5","Type":"ContainerDied","Data":"97f081b630d70ff9210e46ce56423bc8bf42fd50739b1ffae6589a6e2884b6bf"}
Oct 30 22:27:11.789943 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:11.789922    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:27:11.919891 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:11.919858    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kube-rbac-proxy-sar-config\") pod \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") "
Oct 30 22:27:11.920082 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:11.919936    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kserve-provision-location\") pod \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") "
Oct 30 22:27:11.920082 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:11.919963    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kube-api-access-8kqp5\" (UniqueName: \"kubernetes.io/projected/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kube-api-access-8kqp5\") pod \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") "
Oct 30 22:27:11.920082 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:11.919989    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/767f7ae4-f581-4bbe-a746-2cd359ebeff5-proxy-tls\" (UniqueName: \"kubernetes.io/secret/767f7ae4-f581-4bbe-a746-2cd359ebeff5-proxy-tls\") pod \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") "
Oct 30 22:27:11.920082 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:11.920023    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/767f7ae4-f581-4bbe-a746-2cd359ebeff5-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/767f7ae4-f581-4bbe-a746-2cd359ebeff5-cabundle-cert\") pod \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\" (UID: \"767f7ae4-f581-4bbe-a746-2cd359ebeff5\") "
Oct 30 22:27:11.920383 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:11.920268    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kserve-provision-location" pod "767f7ae4-f581-4bbe-a746-2cd359ebeff5" (UID: "767f7ae4-f581-4bbe-a746-2cd359ebeff5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:11.920437 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:11.920302    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kube-rbac-proxy-sar-config" pod "767f7ae4-f581-4bbe-a746-2cd359ebeff5" (UID: "767f7ae4-f581-4bbe-a746-2cd359ebeff5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:11.920482 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:11.920455    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767f7ae4-f581-4bbe-a746-2cd359ebeff5-cabundle-cert" pod "767f7ae4-f581-4bbe-a746-2cd359ebeff5" (UID: "767f7ae4-f581-4bbe-a746-2cd359ebeff5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:11.922282 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:11.922261    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/767f7ae4-f581-4bbe-a746-2cd359ebeff5-proxy-tls" pod "767f7ae4-f581-4bbe-a746-2cd359ebeff5" (UID: "767f7ae4-f581-4bbe-a746-2cd359ebeff5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:11.922424 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:11.922402    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kube-api-access-8kqp5" pod "767f7ae4-f581-4bbe-a746-2cd359ebeff5" (UID: "767f7ae4-f581-4bbe-a746-2cd359ebeff5"). InnerVolumeSpecName "kube-api-access-8kqp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:12.021552 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:12.021460    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.021552 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:12.021501    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kqp5\" (UniqueName: \"kubernetes.io/projected/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kube-api-access-8kqp5\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.021552 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:12.021513    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/767f7ae4-f581-4bbe-a746-2cd359ebeff5-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.021552 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:12.021523    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/767f7ae4-f581-4bbe-a746-2cd359ebeff5-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.021552 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:12.021532    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/767f7ae4-f581-4bbe-a746-2cd359ebeff5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.727139 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:12.727105    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z" event={"ID":"767f7ae4-f581-4bbe-a746-2cd359ebeff5","Type":"ContainerDied","Data":"dcfb9921bfb5260043c4e0f167dc01c6b5cf762a0cc638dcb7b4c14e74133f21"}
Oct 30 22:27:12.727139 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:12.727128    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"
Oct 30 22:27:12.727623 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:12.727177    4947 scope.go:122] "RemoveContainer" containerID="cd873534ead558a9af1650d74d4ce5562008d84f3f08f7ac16c2721a5c3d46c8"
Oct 30 22:27:12.735384 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:12.735366    4947 scope.go:122] "RemoveContainer" containerID="97f081b630d70ff9210e46ce56423bc8bf42fd50739b1ffae6589a6e2884b6bf"
Oct 30 22:27:12.742530 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:12.742514    4947 scope.go:122] "RemoveContainer" containerID="ba36d83858228efb7df0c7617dea6b1a3391af0a7a489109d78414554e6b93f0"
Oct 30 22:27:12.944763 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:12.944731    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"]
Oct 30 22:27:12.994520 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:12.994437    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-xgboost-predictor-7986b66cc6-c5j4z"]
Oct 30 22:27:14.158006 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:14.157976    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" path="/var/lib/kubelet/pods/767f7ae4-f581-4bbe-a746-2cd359ebeff5/volumes"
Oct 30 22:27:15.679075 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.679036    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"]
Oct 30 22:27:15.679486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.679388    4947 cpu_manager.go:414] "containerMap: removing container" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kube-rbac-proxy"
Oct 30 22:27:15.679486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.679403    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kube-rbac-proxy"
Oct 30 22:27:15.679486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.679419    4947 cpu_manager.go:414] "containerMap: removing container" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kserve-container"
Oct 30 22:27:15.679486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.679425    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kserve-container"
Oct 30 22:27:15.679486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.679440    4947 cpu_manager.go:414] "containerMap: removing container" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="storage-initializer"
Oct 30 22:27:15.679486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.679445    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="storage-initializer"
Oct 30 22:27:15.679670 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.679495    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kserve-container"
Oct 30 22:27:15.679670 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.679508    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="767f7ae4-f581-4bbe-a746-2cd359ebeff5" containerName="kube-rbac-proxy"
Oct 30 22:27:15.682255 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.682239    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:15.736392 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.736355    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:27:15.785189 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.785163    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:27:15.785376 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.785237    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Oct 30 22:27:15.785606 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.785586    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:27:15.785673 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.785633    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:27:15.785725 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.785680    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Oct 30 22:27:15.785725 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.785694    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:27:15.787427 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.787404    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"]
Oct 30 22:27:15.852788 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.852748    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:15.852973 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.852796    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzdkx\" (UniqueName: \"kubernetes.io/projected/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kube-api-access-vzdkx\") pod \"isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:15.852973 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.852902    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:15.852973 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.852961    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:15.853117 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.852993    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:15.953961 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.953858    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:15.953961 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.953930    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:15.953961 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.953952    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzdkx\" (UniqueName: \"kubernetes.io/projected/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kube-api-access-vzdkx\") pod \"isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:15.954254 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.953989    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:15.954254 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.954110    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:15.954505 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.954478    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:15.954699 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.954681    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:15.954814 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.954795    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:15.956543 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:15.956514    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:16.020276 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:16.020234    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzdkx\" (UniqueName: \"kubernetes.io/projected/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kube-api-access-vzdkx\") pod \"isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:16.295633 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:16.295600    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:16.721793 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:16.721712    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"]
Oct 30 22:27:16.759223 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:27:16.759185    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8c2c18_38f1_4fbd_a0f1_eff52adcc0cb.slice/crio-72dc6a1c05fec37416834ce29aea711c21ffffe38d55d051becbc86c8cc64511 WatchSource:0}: Error finding container 72dc6a1c05fec37416834ce29aea711c21ffffe38d55d051becbc86c8cc64511: Status 404 returned error can't find the container with id 72dc6a1c05fec37416834ce29aea711c21ffffe38d55d051becbc86c8cc64511
Oct 30 22:27:17.747758 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:17.747719    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb" event={"ID":"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb","Type":"ContainerStarted","Data":"2ec91594eef54b961a2a830034da52b410fb65261238c8b09b84f5d958d4d819"}
Oct 30 22:27:17.748150 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:17.747812    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb" event={"ID":"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb","Type":"ContainerStarted","Data":"72dc6a1c05fec37416834ce29aea711c21ffffe38d55d051becbc86c8cc64511"}
Oct 30 22:27:21.765427 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:21.765394    4947 generic.go:356] "Generic (PLEG): container finished" podID="1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" containerID="2ec91594eef54b961a2a830034da52b410fb65261238c8b09b84f5d958d4d819" exitCode=0
Oct 30 22:27:21.765836 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:21.765469    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb" event={"ID":"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb","Type":"ContainerDied","Data":"2ec91594eef54b961a2a830034da52b410fb65261238c8b09b84f5d958d4d819"}
Oct 30 22:27:22.771362 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:22.771302    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb" event={"ID":"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb","Type":"ContainerStarted","Data":"b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767"}
Oct 30 22:27:22.771753 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:22.771406    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb" event={"ID":"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb","Type":"ContainerStarted","Data":"5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e"}
Oct 30 22:27:22.771753 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:22.771645    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:22.771753 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:22.771688    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:22.773080 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:22.773057    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb" podUID="1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Oct 30 22:27:22.892773 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:22.892697    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb" podStartSLOduration=3.9479607850000003 podStartE2EDuration="7.89267864s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.944717855s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:22.89194098 +0000 UTC m=+1140.891288543" watchObservedRunningTime="2026-10-30 22:27:22.89267864 +0000 UTC m=+1140.892026205"
Oct 30 22:27:23.776530 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:23.776495    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb" podUID="1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Oct 30 22:27:28.781098 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:28.781068    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:28.782226 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:28.782201    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:29.048222 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:27:29.048189    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:27:35.796814 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:35.796778    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"]
Oct 30 22:27:35.797380 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:35.797027    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb" podUID="1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" containerName="kserve-container" containerID="cri-o://5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e" gracePeriod=30
Oct 30 22:27:35.797380 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:35.797083    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb" podUID="1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" containerName="kube-rbac-proxy" containerID="cri-o://b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767" gracePeriod=30
Oct 30 22:27:36.525345 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.525306    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:36.610023 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.609942    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-cabundle-cert\") pod \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") "
Oct 30 22:27:36.610023 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.610000    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-proxy-tls\") pod \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") "
Oct 30 22:27:36.610023 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.610033    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kube-rbac-proxy-sar-config\") pod \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") "
Oct 30 22:27:36.610315 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.610070    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kserve-provision-location\") pod \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") "
Oct 30 22:27:36.610315 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.610164    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kube-api-access-vzdkx\" (UniqueName: \"kubernetes.io/projected/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kube-api-access-vzdkx\") pod \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\" (UID: \"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb\") "
Oct 30 22:27:36.610437 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.610370    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-cabundle-cert" pod "1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" (UID: "1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:36.610486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.610436    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:36.610486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.610454    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kserve-provision-location" pod "1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" (UID: "1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:36.610486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.610454    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kube-rbac-proxy-sar-config" pod "1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" (UID: "1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:36.612370 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.612316    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-proxy-tls" pod "1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" (UID: "1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:36.612491 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.612419    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kube-api-access-vzdkx" pod "1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" (UID: "1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb"). InnerVolumeSpecName "kube-api-access-vzdkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:36.710763 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.710729    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:36.710763 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.710760    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:36.710763 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.710771    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:36.710980 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.710780    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzdkx\" (UniqueName: \"kubernetes.io/projected/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb-kube-api-access-vzdkx\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:36.830610 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.830570    4947 generic.go:356] "Generic (PLEG): container finished" podID="1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" containerID="b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767" exitCode=0
Oct 30 22:27:36.830610 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.830598    4947 generic.go:356] "Generic (PLEG): container finished" podID="1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" containerID="5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e" exitCode=0
Oct 30 22:27:36.831133 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.830652    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"
Oct 30 22:27:36.831133 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.830653    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb" event={"ID":"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb","Type":"ContainerDied","Data":"b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767"}
Oct 30 22:27:36.831133 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.830818    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb" event={"ID":"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb","Type":"ContainerDied","Data":"5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e"}
Oct 30 22:27:36.831133 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.830841    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb" event={"ID":"1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb","Type":"ContainerDied","Data":"72dc6a1c05fec37416834ce29aea711c21ffffe38d55d051becbc86c8cc64511"}
Oct 30 22:27:36.831133 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.830867    4947 scope.go:122] "RemoveContainer" containerID="b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767"
Oct 30 22:27:36.839995 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.839977    4947 scope.go:122] "RemoveContainer" containerID="5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e"
Oct 30 22:27:36.847151 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.847137    4947 scope.go:122] "RemoveContainer" containerID="2ec91594eef54b961a2a830034da52b410fb65261238c8b09b84f5d958d4d819"
Oct 30 22:27:36.854399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.854382    4947 scope.go:122] "RemoveContainer" containerID="b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767"
Oct 30 22:27:36.854694 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:27:36.854670    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767\": container with ID starting with b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767 not found: ID does not exist" containerID="b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767"
Oct 30 22:27:36.854791 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.854701    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767"} err="failed to get container status \"b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767\": rpc error: code = NotFound desc = could not find container \"b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767\": container with ID starting with b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767 not found: ID does not exist"
Oct 30 22:27:36.854791 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.854719    4947 scope.go:122] "RemoveContainer" containerID="5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e"
Oct 30 22:27:36.854972 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:27:36.854950    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e\": container with ID starting with 5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e not found: ID does not exist" containerID="5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e"
Oct 30 22:27:36.855039 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.854983    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e"} err="failed to get container status \"5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e\": rpc error: code = NotFound desc = could not find container \"5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e\": container with ID starting with 5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e not found: ID does not exist"
Oct 30 22:27:36.855039 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.855007    4947 scope.go:122] "RemoveContainer" containerID="2ec91594eef54b961a2a830034da52b410fb65261238c8b09b84f5d958d4d819"
Oct 30 22:27:36.855253 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:27:36.855238    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec91594eef54b961a2a830034da52b410fb65261238c8b09b84f5d958d4d819\": container with ID starting with 2ec91594eef54b961a2a830034da52b410fb65261238c8b09b84f5d958d4d819 not found: ID does not exist" containerID="2ec91594eef54b961a2a830034da52b410fb65261238c8b09b84f5d958d4d819"
Oct 30 22:27:36.855291 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.855257    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec91594eef54b961a2a830034da52b410fb65261238c8b09b84f5d958d4d819"} err="failed to get container status \"2ec91594eef54b961a2a830034da52b410fb65261238c8b09b84f5d958d4d819\": rpc error: code = NotFound desc = could not find container \"2ec91594eef54b961a2a830034da52b410fb65261238c8b09b84f5d958d4d819\": container with ID starting with 2ec91594eef54b961a2a830034da52b410fb65261238c8b09b84f5d958d4d819 not found: ID does not exist"
Oct 30 22:27:36.855291 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.855269    4947 scope.go:122] "RemoveContainer" containerID="b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767"
Oct 30 22:27:36.855517 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.855498    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767"} err="failed to get container status \"b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767\": rpc error: code = NotFound desc = could not find container \"b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767\": container with ID starting with b4e75b2ff333731a666ca79138c58246e780bf5e627b298fbff54f553aef8767 not found: ID does not exist"
Oct 30 22:27:36.855560 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.855517    4947 scope.go:122] "RemoveContainer" containerID="5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e"
Oct 30 22:27:36.855762 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.855745    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e"} err="failed to get container status \"5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e\": rpc error: code = NotFound desc = could not find container \"5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e\": container with ID starting with 5395fa2d031ea9a0a631d7456ba4e00846ccfdc5d10301fe1d6b738483d0176e not found: ID does not exist"
Oct 30 22:27:36.855809 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.855763    4947 scope.go:122] "RemoveContainer" containerID="2ec91594eef54b961a2a830034da52b410fb65261238c8b09b84f5d958d4d819"
Oct 30 22:27:36.855995 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.855979    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec91594eef54b961a2a830034da52b410fb65261238c8b09b84f5d958d4d819"} err="failed to get container status \"2ec91594eef54b961a2a830034da52b410fb65261238c8b09b84f5d958d4d819\": rpc error: code = NotFound desc = could not find container \"2ec91594eef54b961a2a830034da52b410fb65261238c8b09b84f5d958d4d819\": container with ID starting with 2ec91594eef54b961a2a830034da52b410fb65261238c8b09b84f5d958d4d819 not found: ID does not exist"
Oct 30 22:27:36.999550 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:36.999477    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"]
Oct 30 22:27:37.006771 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:37.006742    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-lightgbm-predictor-6c4798b477-nw5lb"]
Oct 30 22:27:38.160641 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:38.160602    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" path="/var/lib/kubelet/pods/1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb/volumes"
Oct 30 22:27:40.786473 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.786444    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"]
Oct 30 22:27:40.786854 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.786745    4947 cpu_manager.go:414] "containerMap: removing container" podUID="1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" containerName="storage-initializer"
Oct 30 22:27:40.786854 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.786755    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" containerName="storage-initializer"
Oct 30 22:27:40.786854 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.786765    4947 cpu_manager.go:414] "containerMap: removing container" podUID="1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" containerName="kserve-container"
Oct 30 22:27:40.786854 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.786769    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" containerName="kserve-container"
Oct 30 22:27:40.786854 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.786784    4947 cpu_manager.go:414] "containerMap: removing container" podUID="1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" containerName="kube-rbac-proxy"
Oct 30 22:27:40.786854 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.786788    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" containerName="kube-rbac-proxy"
Oct 30 22:27:40.786854 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.786846    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" containerName="kube-rbac-proxy"
Oct 30 22:27:40.787059 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.786858    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8c2c18-38f1-4fbd-a0f1-eff52adcc0cb" containerName="kserve-container"
Oct 30 22:27:40.788981 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.788966    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:40.844775 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.844744    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4tbt\" (UniqueName: \"kubernetes.io/projected/707ee893-c7a7-4a67-9c48-db74d37c4241-kube-api-access-q4tbt\") pod \"isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") " pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:40.844957 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.844794    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/707ee893-c7a7-4a67-9c48-db74d37c4241-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") " pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:40.844957 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.844818    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/707ee893-c7a7-4a67-9c48-db74d37c4241-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") " pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:40.844957 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.844901    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/707ee893-c7a7-4a67-9c48-db74d37c4241-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") " pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:40.844957 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.844928    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/707ee893-c7a7-4a67-9c48-db74d37c4241-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") " pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:40.893455 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.893425    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"]
Oct 30 22:27:40.930922 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.930885    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:27:40.931439 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.931423    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:27:40.931533 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.931496    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Oct 30 22:27:40.941951 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.941929    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:27:40.946306 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.946285    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4tbt\" (UniqueName: \"kubernetes.io/projected/707ee893-c7a7-4a67-9c48-db74d37c4241-kube-api-access-q4tbt\") pod \"isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") " pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:40.946390 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.946328    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/707ee893-c7a7-4a67-9c48-db74d37c4241-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") " pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:40.946428 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.946387    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/707ee893-c7a7-4a67-9c48-db74d37c4241-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") " pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:40.946428 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.946417    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/707ee893-c7a7-4a67-9c48-db74d37c4241-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") " pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:40.946506 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.946441    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/707ee893-c7a7-4a67-9c48-db74d37c4241-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") " pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:40.946887 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.946862    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/707ee893-c7a7-4a67-9c48-db74d37c4241-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") " pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:40.947184 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.947164    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/707ee893-c7a7-4a67-9c48-db74d37c4241-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") " pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:40.949098 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.949076    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/707ee893-c7a7-4a67-9c48-db74d37c4241-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") " pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:40.956530 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.956510    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:27:40.956530 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.956515    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Oct 30 22:27:40.956659 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.956535    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:27:40.957931 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:40.957914    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/707ee893-c7a7-4a67-9c48-db74d37c4241-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") " pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:41.008969 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:41.008933    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4tbt\" (UniqueName: \"kubernetes.io/projected/707ee893-c7a7-4a67-9c48-db74d37c4241-kube-api-access-q4tbt\") pod \"isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") " pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:41.103511 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:41.103424    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:41.527447 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:41.527368    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"]
Oct 30 22:27:41.530355 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:27:41.530311    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707ee893_c7a7_4a67_9c48_db74d37c4241.slice/crio-3f723594c7fd609ed9ebfd64c95c5feadc1bf899b7ac9507215f723ff83ec64b WatchSource:0}: Error finding container 3f723594c7fd609ed9ebfd64c95c5feadc1bf899b7ac9507215f723ff83ec64b: Status 404 returned error can't find the container with id 3f723594c7fd609ed9ebfd64c95c5feadc1bf899b7ac9507215f723ff83ec64b
Oct 30 22:27:41.856295 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:41.856256    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9" event={"ID":"707ee893-c7a7-4a67-9c48-db74d37c4241","Type":"ContainerStarted","Data":"87dd8727153be430f5c60d91a1f0d75ee5b0381a682106a48e709a7433d45f1e"}
Oct 30 22:27:41.856801 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:41.856315    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9" event={"ID":"707ee893-c7a7-4a67-9c48-db74d37c4241","Type":"ContainerStarted","Data":"3f723594c7fd609ed9ebfd64c95c5feadc1bf899b7ac9507215f723ff83ec64b"}
Oct 30 22:27:45.874170 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:45.874132    4947 generic.go:356] "Generic (PLEG): container finished" podID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerID="87dd8727153be430f5c60d91a1f0d75ee5b0381a682106a48e709a7433d45f1e" exitCode=0
Oct 30 22:27:45.874623 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:45.874203    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9" event={"ID":"707ee893-c7a7-4a67-9c48-db74d37c4241","Type":"ContainerDied","Data":"87dd8727153be430f5c60d91a1f0d75ee5b0381a682106a48e709a7433d45f1e"}
Oct 30 22:27:46.880558 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:46.880527    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9" event={"ID":"707ee893-c7a7-4a67-9c48-db74d37c4241","Type":"ContainerStarted","Data":"69317e41ab2ca21db2e755864e3cb1da6f15ba3cfeefa2633b8b6e809175aea5"}
Oct 30 22:27:46.880939 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:46.880584    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9" event={"ID":"707ee893-c7a7-4a67-9c48-db74d37c4241","Type":"ContainerStarted","Data":"961c2db2b778b6f4a33e3fcd5f0e566bd6558c895a146ca3962606c62152ae76"}
Oct 30 22:27:46.880939 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:46.880776    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:47.002237 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:47.002180    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9" podStartSLOduration=2.918443892 podStartE2EDuration="7.002164422s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.08372053s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:47.001366161 +0000 UTC m=+1165.000713729" watchObservedRunningTime="2026-10-30 22:27:47.002164422 +0000 UTC m=+1165.001511985"
Oct 30 22:27:47.884671 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:47.884635    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:53.895481 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:53.895449    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:27:59.048597 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:27:59.048562    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:27:59.935654 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:59.935617    4947 generic.go:356] "Generic (PLEG): container finished" podID="79347464-bcf5-4fc4-95d7-6e6c406d0332" containerID="0994c2e944b2d5a9c3d7fec8c98faf87db54c3d15b77f79c4cce6ea9c7958f84" exitCode=0
Oct 30 22:27:59.935853 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:27:59.935698    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5mdb7-fg8fs" event={"ID":"79347464-bcf5-4fc4-95d7-6e6c406d0332","Type":"ContainerDied","Data":"0994c2e944b2d5a9c3d7fec8c98faf87db54c3d15b77f79c4cce6ea9c7958f84"}
Oct 30 22:28:01.070293 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:01.070269    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5mdb7-fg8fs"
Oct 30 22:28:01.118009 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:01.117967    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79347464-bcf5-4fc4-95d7-6e6c406d0332-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79347464-bcf5-4fc4-95d7-6e6c406d0332-trusted-ca-bundle\") pod \"79347464-bcf5-4fc4-95d7-6e6c406d0332\" (UID: \"79347464-bcf5-4fc4-95d7-6e6c406d0332\") "
Oct 30 22:28:01.118182 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:01.118068    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79347464-bcf5-4fc4-95d7-6e6c406d0332-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/79347464-bcf5-4fc4-95d7-6e6c406d0332-archives-path\") pod \"79347464-bcf5-4fc4-95d7-6e6c406d0332\" (UID: \"79347464-bcf5-4fc4-95d7-6e6c406d0332\") "
Oct 30 22:28:01.118182 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:01.118144    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79347464-bcf5-4fc4-95d7-6e6c406d0332-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79347464-bcf5-4fc4-95d7-6e6c406d0332-service-ca-bundle\") pod \"79347464-bcf5-4fc4-95d7-6e6c406d0332\" (UID: \"79347464-bcf5-4fc4-95d7-6e6c406d0332\") "
Oct 30 22:28:01.118298 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:01.118188    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79347464-bcf5-4fc4-95d7-6e6c406d0332-kube-api-access-28h64\" (UniqueName: \"kubernetes.io/projected/79347464-bcf5-4fc4-95d7-6e6c406d0332-kube-api-access-28h64\") pod \"79347464-bcf5-4fc4-95d7-6e6c406d0332\" (UID: \"79347464-bcf5-4fc4-95d7-6e6c406d0332\") "
Oct 30 22:28:01.118298 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:01.118261    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79347464-bcf5-4fc4-95d7-6e6c406d0332-trusted-ca-bundle" pod "79347464-bcf5-4fc4-95d7-6e6c406d0332" (UID: "79347464-bcf5-4fc4-95d7-6e6c406d0332"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:01.118472 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:01.118447    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79347464-bcf5-4fc4-95d7-6e6c406d0332-archives-path" pod "79347464-bcf5-4fc4-95d7-6e6c406d0332" (UID: "79347464-bcf5-4fc4-95d7-6e6c406d0332"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:01.118509 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:01.118475    4947 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79347464-bcf5-4fc4-95d7-6e6c406d0332-trusted-ca-bundle\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:01.118509 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:01.118499    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79347464-bcf5-4fc4-95d7-6e6c406d0332-service-ca-bundle" pod "79347464-bcf5-4fc4-95d7-6e6c406d0332" (UID: "79347464-bcf5-4fc4-95d7-6e6c406d0332"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:01.120437 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:01.120420    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79347464-bcf5-4fc4-95d7-6e6c406d0332-kube-api-access-28h64" pod "79347464-bcf5-4fc4-95d7-6e6c406d0332" (UID: "79347464-bcf5-4fc4-95d7-6e6c406d0332"). InnerVolumeSpecName "kube-api-access-28h64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:01.219501 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:01.219427    4947 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79347464-bcf5-4fc4-95d7-6e6c406d0332-service-ca-bundle\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:01.219501 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:01.219458    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28h64\" (UniqueName: \"kubernetes.io/projected/79347464-bcf5-4fc4-95d7-6e6c406d0332-kube-api-access-28h64\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:01.219501 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:01.219469    4947 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/79347464-bcf5-4fc4-95d7-6e6c406d0332-archives-path\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:01.944931 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:01.944894    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5mdb7-fg8fs" event={"ID":"79347464-bcf5-4fc4-95d7-6e6c406d0332","Type":"ContainerDied","Data":"76af867ec18d33806a629679215cb0580de224e2b2f17a58cd0d1b4dbcd68531"}
Oct 30 22:28:01.945146 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:01.944986    4947 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76af867ec18d33806a629679215cb0580de224e2b2f17a58cd0d1b4dbcd68531"
Oct 30 22:28:01.945146 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:01.944913    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5mdb7-fg8fs"
Oct 30 22:28:22.251292 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:22.251260    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:28:22.257725 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:22.257706    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:28:23.896920 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:23.896883    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9" podUID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.39:8080: connect: connection refused"
Oct 30 22:28:29.047935 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:28:29.047897    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:28:33.896921 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:33.896825    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9" podUID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.39:8080: connect: connection refused"
Oct 30 22:28:43.896643 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:43.896597    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9" podUID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.39:8080: connect: connection refused"
Oct 30 22:28:54.405574 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:28:54.405545    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:28:59.047800 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:28:59.047764    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:29:00.961137 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:00.961100    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"]
Oct 30 22:29:00.961586 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:00.961365    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9" podUID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerName="kserve-container" containerID="cri-o://961c2db2b778b6f4a33e3fcd5f0e566bd6558c895a146ca3962606c62152ae76" gracePeriod=30
Oct 30 22:29:00.961586 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:00.961427    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9" podUID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerName="kube-rbac-proxy" containerID="cri-o://69317e41ab2ca21db2e755864e3cb1da6f15ba3cfeefa2633b8b6e809175aea5" gracePeriod=30
Oct 30 22:29:01.185788 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:01.185750    4947 generic.go:356] "Generic (PLEG): container finished" podID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerID="69317e41ab2ca21db2e755864e3cb1da6f15ba3cfeefa2633b8b6e809175aea5" exitCode=0
Oct 30 22:29:01.185950 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:01.185825    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9" event={"ID":"707ee893-c7a7-4a67-9c48-db74d37c4241","Type":"ContainerDied","Data":"69317e41ab2ca21db2e755864e3cb1da6f15ba3cfeefa2633b8b6e809175aea5"}
Oct 30 22:29:03.890377 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:03.890307    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9" podUID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Oct 30 22:29:03.896900 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:03.896864    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9" podUID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.39:8080: connect: connection refused"
Oct 30 22:29:05.611004 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:05.610979    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:29:05.745515 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:05.745423    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/707ee893-c7a7-4a67-9c48-db74d37c4241-kube-api-access-q4tbt\" (UniqueName: \"kubernetes.io/projected/707ee893-c7a7-4a67-9c48-db74d37c4241-kube-api-access-q4tbt\") pod \"707ee893-c7a7-4a67-9c48-db74d37c4241\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") "
Oct 30 22:29:05.745515 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:05.745499    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/707ee893-c7a7-4a67-9c48-db74d37c4241-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/707ee893-c7a7-4a67-9c48-db74d37c4241-kube-rbac-proxy-sar-config\") pod \"707ee893-c7a7-4a67-9c48-db74d37c4241\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") "
Oct 30 22:29:05.745749 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:05.745568    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707ee893-c7a7-4a67-9c48-db74d37c4241-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/707ee893-c7a7-4a67-9c48-db74d37c4241-kserve-provision-location\") pod \"707ee893-c7a7-4a67-9c48-db74d37c4241\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") "
Oct 30 22:29:05.745749 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:05.745591    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/707ee893-c7a7-4a67-9c48-db74d37c4241-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/707ee893-c7a7-4a67-9c48-db74d37c4241-cabundle-cert\") pod \"707ee893-c7a7-4a67-9c48-db74d37c4241\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") "
Oct 30 22:29:05.745749 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:05.745615    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/707ee893-c7a7-4a67-9c48-db74d37c4241-proxy-tls\" (UniqueName: \"kubernetes.io/secret/707ee893-c7a7-4a67-9c48-db74d37c4241-proxy-tls\") pod \"707ee893-c7a7-4a67-9c48-db74d37c4241\" (UID: \"707ee893-c7a7-4a67-9c48-db74d37c4241\") "
Oct 30 22:29:05.745911 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:05.745891    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707ee893-c7a7-4a67-9c48-db74d37c4241-kserve-provision-location" pod "707ee893-c7a7-4a67-9c48-db74d37c4241" (UID: "707ee893-c7a7-4a67-9c48-db74d37c4241"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:05.745964 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:05.745921    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707ee893-c7a7-4a67-9c48-db74d37c4241-kube-rbac-proxy-sar-config" pod "707ee893-c7a7-4a67-9c48-db74d37c4241" (UID: "707ee893-c7a7-4a67-9c48-db74d37c4241"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:05.746002 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:05.745964    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707ee893-c7a7-4a67-9c48-db74d37c4241-cabundle-cert" pod "707ee893-c7a7-4a67-9c48-db74d37c4241" (UID: "707ee893-c7a7-4a67-9c48-db74d37c4241"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:05.747685 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:05.747661    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707ee893-c7a7-4a67-9c48-db74d37c4241-kube-api-access-q4tbt" pod "707ee893-c7a7-4a67-9c48-db74d37c4241" (UID: "707ee893-c7a7-4a67-9c48-db74d37c4241"). InnerVolumeSpecName "kube-api-access-q4tbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:05.747812 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:05.747740    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/707ee893-c7a7-4a67-9c48-db74d37c4241-proxy-tls" pod "707ee893-c7a7-4a67-9c48-db74d37c4241" (UID: "707ee893-c7a7-4a67-9c48-db74d37c4241"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:05.846505 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:05.846473    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/707ee893-c7a7-4a67-9c48-db74d37c4241-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:05.846505 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:05.846499    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/707ee893-c7a7-4a67-9c48-db74d37c4241-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:05.846505 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:05.846508    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/707ee893-c7a7-4a67-9c48-db74d37c4241-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:05.846729 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:05.846517    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4tbt\" (UniqueName: \"kubernetes.io/projected/707ee893-c7a7-4a67-9c48-db74d37c4241-kube-api-access-q4tbt\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:05.846729 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:05.846527    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/707ee893-c7a7-4a67-9c48-db74d37c4241-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:29:06.208103 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:06.208069    4947 generic.go:356] "Generic (PLEG): container finished" podID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerID="961c2db2b778b6f4a33e3fcd5f0e566bd6558c895a146ca3962606c62152ae76" exitCode=0
Oct 30 22:29:06.208256 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:06.208135    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"
Oct 30 22:29:06.208256 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:06.208132    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9" event={"ID":"707ee893-c7a7-4a67-9c48-db74d37c4241","Type":"ContainerDied","Data":"961c2db2b778b6f4a33e3fcd5f0e566bd6558c895a146ca3962606c62152ae76"}
Oct 30 22:29:06.208325 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:06.208274    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9" event={"ID":"707ee893-c7a7-4a67-9c48-db74d37c4241","Type":"ContainerDied","Data":"3f723594c7fd609ed9ebfd64c95c5feadc1bf899b7ac9507215f723ff83ec64b"}
Oct 30 22:29:06.208325 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:06.208301    4947 scope.go:122] "RemoveContainer" containerID="69317e41ab2ca21db2e755864e3cb1da6f15ba3cfeefa2633b8b6e809175aea5"
Oct 30 22:29:06.216406 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:06.216388    4947 scope.go:122] "RemoveContainer" containerID="961c2db2b778b6f4a33e3fcd5f0e566bd6558c895a146ca3962606c62152ae76"
Oct 30 22:29:06.223531 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:06.223515    4947 scope.go:122] "RemoveContainer" containerID="87dd8727153be430f5c60d91a1f0d75ee5b0381a682106a48e709a7433d45f1e"
Oct 30 22:29:06.230536 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:06.230520    4947 scope.go:122] "RemoveContainer" containerID="69317e41ab2ca21db2e755864e3cb1da6f15ba3cfeefa2633b8b6e809175aea5"
Oct 30 22:29:06.230831 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:29:06.230812    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69317e41ab2ca21db2e755864e3cb1da6f15ba3cfeefa2633b8b6e809175aea5\": container with ID starting with 69317e41ab2ca21db2e755864e3cb1da6f15ba3cfeefa2633b8b6e809175aea5 not found: ID does not exist" containerID="69317e41ab2ca21db2e755864e3cb1da6f15ba3cfeefa2633b8b6e809175aea5"
Oct 30 22:29:06.230896 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:06.230842    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69317e41ab2ca21db2e755864e3cb1da6f15ba3cfeefa2633b8b6e809175aea5"} err="failed to get container status \"69317e41ab2ca21db2e755864e3cb1da6f15ba3cfeefa2633b8b6e809175aea5\": rpc error: code = NotFound desc = could not find container \"69317e41ab2ca21db2e755864e3cb1da6f15ba3cfeefa2633b8b6e809175aea5\": container with ID starting with 69317e41ab2ca21db2e755864e3cb1da6f15ba3cfeefa2633b8b6e809175aea5 not found: ID does not exist"
Oct 30 22:29:06.230896 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:06.230864    4947 scope.go:122] "RemoveContainer" containerID="961c2db2b778b6f4a33e3fcd5f0e566bd6558c895a146ca3962606c62152ae76"
Oct 30 22:29:06.231109 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:29:06.231092    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961c2db2b778b6f4a33e3fcd5f0e566bd6558c895a146ca3962606c62152ae76\": container with ID starting with 961c2db2b778b6f4a33e3fcd5f0e566bd6558c895a146ca3962606c62152ae76 not found: ID does not exist" containerID="961c2db2b778b6f4a33e3fcd5f0e566bd6558c895a146ca3962606c62152ae76"
Oct 30 22:29:06.231170 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:06.231118    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961c2db2b778b6f4a33e3fcd5f0e566bd6558c895a146ca3962606c62152ae76"} err="failed to get container status \"961c2db2b778b6f4a33e3fcd5f0e566bd6558c895a146ca3962606c62152ae76\": rpc error: code = NotFound desc = could not find container \"961c2db2b778b6f4a33e3fcd5f0e566bd6558c895a146ca3962606c62152ae76\": container with ID starting with 961c2db2b778b6f4a33e3fcd5f0e566bd6558c895a146ca3962606c62152ae76 not found: ID does not exist"
Oct 30 22:29:06.231170 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:06.231140    4947 scope.go:122] "RemoveContainer" containerID="87dd8727153be430f5c60d91a1f0d75ee5b0381a682106a48e709a7433d45f1e"
Oct 30 22:29:06.231400 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:29:06.231383    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87dd8727153be430f5c60d91a1f0d75ee5b0381a682106a48e709a7433d45f1e\": container with ID starting with 87dd8727153be430f5c60d91a1f0d75ee5b0381a682106a48e709a7433d45f1e not found: ID does not exist" containerID="87dd8727153be430f5c60d91a1f0d75ee5b0381a682106a48e709a7433d45f1e"
Oct 30 22:29:06.231449 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:06.231405    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87dd8727153be430f5c60d91a1f0d75ee5b0381a682106a48e709a7433d45f1e"} err="failed to get container status \"87dd8727153be430f5c60d91a1f0d75ee5b0381a682106a48e709a7433d45f1e\": rpc error: code = NotFound desc = could not find container \"87dd8727153be430f5c60d91a1f0d75ee5b0381a682106a48e709a7433d45f1e\": container with ID starting with 87dd8727153be430f5c60d91a1f0d75ee5b0381a682106a48e709a7433d45f1e not found: ID does not exist"
Oct 30 22:29:06.376287 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:06.376255    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"]
Oct 30 22:29:06.425628 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:06.425551    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-sklearn-v2-predictor-79b994cfdf-nm7z9"]
Oct 30 22:29:08.158419 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:08.158387    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="707ee893-c7a7-4a67-9c48-db74d37c4241" path="/var/lib/kubelet/pods/707ee893-c7a7-4a67-9c48-db74d37c4241/volumes"
Oct 30 22:29:10.075200 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.075162    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"]
Oct 30 22:29:10.075635 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.075518    4947 cpu_manager.go:414] "containerMap: removing container" podUID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerName="storage-initializer"
Oct 30 22:29:10.075635 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.075532    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerName="storage-initializer"
Oct 30 22:29:10.075635 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.075547    4947 cpu_manager.go:414] "containerMap: removing container" podUID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerName="kube-rbac-proxy"
Oct 30 22:29:10.075635 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.075552    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerName="kube-rbac-proxy"
Oct 30 22:29:10.075635 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.075576    4947 cpu_manager.go:414] "containerMap: removing container" podUID="79347464-bcf5-4fc4-95d7-6e6c406d0332" containerName="insights-gathering"
Oct 30 22:29:10.075635 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.075581    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79347464-bcf5-4fc4-95d7-6e6c406d0332" containerName="insights-gathering"
Oct 30 22:29:10.075635 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.075588    4947 cpu_manager.go:414] "containerMap: removing container" podUID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerName="kserve-container"
Oct 30 22:29:10.075635 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.075592    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerName="kserve-container"
Oct 30 22:29:10.075635 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.075636    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="79347464-bcf5-4fc4-95d7-6e6c406d0332" containerName="insights-gathering"
Oct 30 22:29:10.075947 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.075644    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerName="kserve-container"
Oct 30 22:29:10.075947 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.075652    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="707ee893-c7a7-4a67-9c48-db74d37c4241" containerName="kube-rbac-proxy"
Oct 30 22:29:10.078747 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.078730    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:10.181697 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.181661    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:29:10.181882 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.181711    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Oct 30 22:29:10.181882 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.181757    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:29:10.181882 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.181763    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:29:10.181882 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.181780    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:29:10.182077 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.181942    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:29:10.182077 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.181988    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Oct 30 22:29:10.184219 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.184192    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"]
Oct 30 22:29:10.184354 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.184269    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e382852d-1106-4721-94a3-9de1b3737137-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:10.184424 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.184365    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e382852d-1106-4721-94a3-9de1b3737137-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:10.184482 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.184463    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e382852d-1106-4721-94a3-9de1b3737137-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:10.184551 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.184533    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e382852d-1106-4721-94a3-9de1b3737137-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:10.184793 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.184569    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sdl6\" (UniqueName: \"kubernetes.io/projected/e382852d-1106-4721-94a3-9de1b3737137-kube-api-access-2sdl6\") pod \"isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:10.285483 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.285448    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e382852d-1106-4721-94a3-9de1b3737137-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:10.285483 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.285481    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sdl6\" (UniqueName: \"kubernetes.io/projected/e382852d-1106-4721-94a3-9de1b3737137-kube-api-access-2sdl6\") pod \"isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:10.285705 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.285507    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e382852d-1106-4721-94a3-9de1b3737137-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:10.285705 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.285674    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e382852d-1106-4721-94a3-9de1b3737137-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:10.285814 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.285749    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e382852d-1106-4721-94a3-9de1b3737137-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:10.285960 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.285935    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e382852d-1106-4721-94a3-9de1b3737137-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:10.286475 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.286363    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e382852d-1106-4721-94a3-9de1b3737137-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:10.286475 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.286398    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e382852d-1106-4721-94a3-9de1b3737137-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:10.288150 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.288129    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e382852d-1106-4721-94a3-9de1b3737137-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:10.347886 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.347817    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sdl6\" (UniqueName: \"kubernetes.io/projected/e382852d-1106-4721-94a3-9de1b3737137-kube-api-access-2sdl6\") pod \"isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") " pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:10.395852 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.395818    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:10.820840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:10.820768    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"]
Oct 30 22:29:10.824989 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:29:10.824967    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode382852d_1106_4721_94a3_9de1b3737137.slice/crio-3bfcc7c8ef52b6324f6638661dc62afc97d09d264e1b9e85d6fcd09b6bae6eb9 WatchSource:0}: Error finding container 3bfcc7c8ef52b6324f6638661dc62afc97d09d264e1b9e85d6fcd09b6bae6eb9: Status 404 returned error can't find the container with id 3bfcc7c8ef52b6324f6638661dc62afc97d09d264e1b9e85d6fcd09b6bae6eb9
Oct 30 22:29:11.230446 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:11.230359    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf" event={"ID":"e382852d-1106-4721-94a3-9de1b3737137","Type":"ContainerStarted","Data":"d265c11346ec23a28b4f9fb654bfe21f1b16ab4530f87ea41025007ed287b64b"}
Oct 30 22:29:11.230446 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:11.230439    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf" event={"ID":"e382852d-1106-4721-94a3-9de1b3737137","Type":"ContainerStarted","Data":"3bfcc7c8ef52b6324f6638661dc62afc97d09d264e1b9e85d6fcd09b6bae6eb9"}
Oct 30 22:29:15.247262 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:15.247183    4947 generic.go:356] "Generic (PLEG): container finished" podID="e382852d-1106-4721-94a3-9de1b3737137" containerID="d265c11346ec23a28b4f9fb654bfe21f1b16ab4530f87ea41025007ed287b64b" exitCode=0
Oct 30 22:29:15.247262 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:15.247245    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf" event={"ID":"e382852d-1106-4721-94a3-9de1b3737137","Type":"ContainerDied","Data":"d265c11346ec23a28b4f9fb654bfe21f1b16ab4530f87ea41025007ed287b64b"}
Oct 30 22:29:16.253388 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:16.253356    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf" event={"ID":"e382852d-1106-4721-94a3-9de1b3737137","Type":"ContainerStarted","Data":"51cf1b9635402b4349ac719983d5d84f3537362335dcecc09f1252ebb4c82aad"}
Oct 30 22:29:16.253772 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:16.253424    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf" event={"ID":"e382852d-1106-4721-94a3-9de1b3737137","Type":"ContainerStarted","Data":"acc9959c24e10f6d208f7495ab20b03eb15427c457e038d46b41540febc811a2"}
Oct 30 22:29:16.253772 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:16.253619    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:16.378762 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:16.378698    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf" podStartSLOduration=2.458050713 podStartE2EDuration="6.378685065s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.920634352s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:16.378299618 +0000 UTC m=+1254.377647182" watchObservedRunningTime="2026-10-30 22:29:16.378685065 +0000 UTC m=+1254.378032627"
Oct 30 22:29:17.257882 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:17.257852    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:23.267260 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:23.267231    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:29:29.048809 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:29:29.048775    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:29:53.268872 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:29:53.268833    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf" podUID="e382852d-1106-4721-94a3-9de1b3737137" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.40:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.40:8080: connect: connection refused"
Oct 30 22:29:59.047699 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:29:59.047658    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:30:03.268306 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:03.268222    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf" podUID="e382852d-1106-4721-94a3-9de1b3737137" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.40:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.40:8080: connect: connection refused"
Oct 30 22:30:13.268184 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:13.268146    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf" podUID="e382852d-1106-4721-94a3-9de1b3737137" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.40:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.40:8080: connect: connection refused"
Oct 30 22:30:23.796772 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:23.796735    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:30:29.048976 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:30:29.048927    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:30:30.238779 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:30.238744    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"]
Oct 30 22:30:30.239270 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:30.238996    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf" podUID="e382852d-1106-4721-94a3-9de1b3737137" containerName="kserve-container" containerID="cri-o://acc9959c24e10f6d208f7495ab20b03eb15427c457e038d46b41540febc811a2" gracePeriod=30
Oct 30 22:30:30.239270 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:30.239063    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf" podUID="e382852d-1106-4721-94a3-9de1b3737137" containerName="kube-rbac-proxy" containerID="cri-o://51cf1b9635402b4349ac719983d5d84f3537362335dcecc09f1252ebb4c82aad" gracePeriod=30
Oct 30 22:30:30.552783 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:30.552753    4947 generic.go:356] "Generic (PLEG): container finished" podID="e382852d-1106-4721-94a3-9de1b3737137" containerID="51cf1b9635402b4349ac719983d5d84f3537362335dcecc09f1252ebb4c82aad" exitCode=0
Oct 30 22:30:30.552937 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:30.552832    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf" event={"ID":"e382852d-1106-4721-94a3-9de1b3737137","Type":"ContainerDied","Data":"51cf1b9635402b4349ac719983d5d84f3537362335dcecc09f1252ebb4c82aad"}
Oct 30 22:30:33.262491 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:33.262452    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf" podUID="e382852d-1106-4721-94a3-9de1b3737137" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Oct 30 22:30:33.267880 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:33.267856    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf" podUID="e382852d-1106-4721-94a3-9de1b3737137" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.40:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.40:8080: connect: connection refused"
Oct 30 22:30:34.571733 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.571693    4947 generic.go:356] "Generic (PLEG): container finished" podID="e382852d-1106-4721-94a3-9de1b3737137" containerID="acc9959c24e10f6d208f7495ab20b03eb15427c457e038d46b41540febc811a2" exitCode=0
Oct 30 22:30:34.572060 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.571741    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf" event={"ID":"e382852d-1106-4721-94a3-9de1b3737137","Type":"ContainerDied","Data":"acc9959c24e10f6d208f7495ab20b03eb15427c457e038d46b41540febc811a2"}
Oct 30 22:30:34.686953 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.686930    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:30:34.736933 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.736901    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e382852d-1106-4721-94a3-9de1b3737137-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e382852d-1106-4721-94a3-9de1b3737137-proxy-tls\") pod \"e382852d-1106-4721-94a3-9de1b3737137\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") "
Oct 30 22:30:34.737094 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.736960    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e382852d-1106-4721-94a3-9de1b3737137-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e382852d-1106-4721-94a3-9de1b3737137-kserve-provision-location\") pod \"e382852d-1106-4721-94a3-9de1b3737137\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") "
Oct 30 22:30:34.737094 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.737011    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e382852d-1106-4721-94a3-9de1b3737137-kube-api-access-2sdl6\" (UniqueName: \"kubernetes.io/projected/e382852d-1106-4721-94a3-9de1b3737137-kube-api-access-2sdl6\") pod \"e382852d-1106-4721-94a3-9de1b3737137\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") "
Oct 30 22:30:34.737094 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.737075    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e382852d-1106-4721-94a3-9de1b3737137-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e382852d-1106-4721-94a3-9de1b3737137-kube-rbac-proxy-sar-config\") pod \"e382852d-1106-4721-94a3-9de1b3737137\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") "
Oct 30 22:30:34.737255 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.737100    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e382852d-1106-4721-94a3-9de1b3737137-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e382852d-1106-4721-94a3-9de1b3737137-cabundle-cert\") pod \"e382852d-1106-4721-94a3-9de1b3737137\" (UID: \"e382852d-1106-4721-94a3-9de1b3737137\") "
Oct 30 22:30:34.737255 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.737199    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e382852d-1106-4721-94a3-9de1b3737137-kserve-provision-location" pod "e382852d-1106-4721-94a3-9de1b3737137" (UID: "e382852d-1106-4721-94a3-9de1b3737137"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:34.737434 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.737417    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e382852d-1106-4721-94a3-9de1b3737137-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:34.737487 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.737453    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e382852d-1106-4721-94a3-9de1b3737137-kube-rbac-proxy-sar-config" pod "e382852d-1106-4721-94a3-9de1b3737137" (UID: "e382852d-1106-4721-94a3-9de1b3737137"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:34.737540 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.737517    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e382852d-1106-4721-94a3-9de1b3737137-cabundle-cert" pod "e382852d-1106-4721-94a3-9de1b3737137" (UID: "e382852d-1106-4721-94a3-9de1b3737137"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:34.739078 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.739058    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e382852d-1106-4721-94a3-9de1b3737137-proxy-tls" pod "e382852d-1106-4721-94a3-9de1b3737137" (UID: "e382852d-1106-4721-94a3-9de1b3737137"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:34.739212 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.739194    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e382852d-1106-4721-94a3-9de1b3737137-kube-api-access-2sdl6" pod "e382852d-1106-4721-94a3-9de1b3737137" (UID: "e382852d-1106-4721-94a3-9de1b3737137"). InnerVolumeSpecName "kube-api-access-2sdl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:34.838190 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.838155    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e382852d-1106-4721-94a3-9de1b3737137-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:34.838190 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.838187    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e382852d-1106-4721-94a3-9de1b3737137-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:34.838423 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.838201    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e382852d-1106-4721-94a3-9de1b3737137-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:34.838423 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:34.838215    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sdl6\" (UniqueName: \"kubernetes.io/projected/e382852d-1106-4721-94a3-9de1b3737137-kube-api-access-2sdl6\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:35.577977 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:35.577937    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf" event={"ID":"e382852d-1106-4721-94a3-9de1b3737137","Type":"ContainerDied","Data":"3bfcc7c8ef52b6324f6638661dc62afc97d09d264e1b9e85d6fcd09b6bae6eb9"}
Oct 30 22:30:35.577977 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:35.577979    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"
Oct 30 22:30:35.578488 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:35.578021    4947 scope.go:122] "RemoveContainer" containerID="51cf1b9635402b4349ac719983d5d84f3537362335dcecc09f1252ebb4c82aad"
Oct 30 22:30:35.586946 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:35.586931    4947 scope.go:122] "RemoveContainer" containerID="acc9959c24e10f6d208f7495ab20b03eb15427c457e038d46b41540febc811a2"
Oct 30 22:30:35.594023 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:35.594005    4947 scope.go:122] "RemoveContainer" containerID="d265c11346ec23a28b4f9fb654bfe21f1b16ab4530f87ea41025007ed287b64b"
Oct 30 22:30:35.795918 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:35.795874    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"]
Oct 30 22:30:35.845774 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:35.845706    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-xgboost-v2-predictor-745877bc78-8kbkf"]
Oct 30 22:30:36.158387 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:36.158312    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e382852d-1106-4721-94a3-9de1b3737137" path="/var/lib/kubelet/pods/e382852d-1106-4721-94a3-9de1b3737137/volumes"
Oct 30 22:30:39.316771 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.316744    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"]
Oct 30 22:30:39.317127 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.317062    4947 cpu_manager.go:414] "containerMap: removing container" podUID="e382852d-1106-4721-94a3-9de1b3737137" containerName="kserve-container"
Oct 30 22:30:39.317127 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.317073    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e382852d-1106-4721-94a3-9de1b3737137" containerName="kserve-container"
Oct 30 22:30:39.317127 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.317082    4947 cpu_manager.go:414] "containerMap: removing container" podUID="e382852d-1106-4721-94a3-9de1b3737137" containerName="kube-rbac-proxy"
Oct 30 22:30:39.317127 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.317087    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e382852d-1106-4721-94a3-9de1b3737137" containerName="kube-rbac-proxy"
Oct 30 22:30:39.317127 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.317105    4947 cpu_manager.go:414] "containerMap: removing container" podUID="e382852d-1106-4721-94a3-9de1b3737137" containerName="storage-initializer"
Oct 30 22:30:39.317127 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.317110    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e382852d-1106-4721-94a3-9de1b3737137" containerName="storage-initializer"
Oct 30 22:30:39.317314 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.317170    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e382852d-1106-4721-94a3-9de1b3737137" containerName="kserve-container"
Oct 30 22:30:39.317314 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.317178    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e382852d-1106-4721-94a3-9de1b3737137" containerName="kube-rbac-proxy"
Oct 30 22:30:39.321670 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.321652    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:39.375948 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.375915    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:30:39.425389 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.425318    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Oct 30 22:30:39.425389 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.425363    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:30:39.425569 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.425406    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:30:39.425569 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.425487    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:30:39.425649 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.425573    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:30:39.425690 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.425669    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Oct 30 22:30:39.427006 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.426986    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"]
Oct 30 22:30:39.475556 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.475524    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:39.475763 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.475574    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:39.475763 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.475688    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnxsc\" (UniqueName: \"kubernetes.io/projected/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kube-api-access-lnxsc\") pod \"isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:39.475763 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.475721    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:39.475763 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.475742    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:39.576934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.576850    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:39.576934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.576901    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:39.577154 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.576964    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnxsc\" (UniqueName: \"kubernetes.io/projected/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kube-api-access-lnxsc\") pod \"isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:39.577154 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.576993    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:39.577154 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.577021    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:39.577521 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.577499    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:39.577708 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.577683    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:39.577782 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.577760    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:39.579514 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.579493    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:39.640012 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.639984    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnxsc\" (UniqueName: \"kubernetes.io/projected/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kube-api-access-lnxsc\") pod \"isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") " pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:39.938632 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:39.938541    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:40.365126 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:40.365094    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"]
Oct 30 22:30:40.368551 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:30:40.368527    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a33f9a_8b8f_4de7_8d73_95e6097bbcdd.slice/crio-b2b34ccc79c9d3f1289c40dae038d192a4e3e42040f3a1067ccc33164b5a31b5 WatchSource:0}: Error finding container b2b34ccc79c9d3f1289c40dae038d192a4e3e42040f3a1067ccc33164b5a31b5: Status 404 returned error can't find the container with id b2b34ccc79c9d3f1289c40dae038d192a4e3e42040f3a1067ccc33164b5a31b5
Oct 30 22:30:40.601631 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:40.601596    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz" event={"ID":"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd","Type":"ContainerStarted","Data":"53cf1fb407f442abb431c7f51ae8ac8e746eef748683e72a83ecaf8640223707"}
Oct 30 22:30:40.601799 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:40.601672    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz" event={"ID":"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd","Type":"ContainerStarted","Data":"b2b34ccc79c9d3f1289c40dae038d192a4e3e42040f3a1067ccc33164b5a31b5"}
Oct 30 22:30:44.618401 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:44.618370    4947 generic.go:356] "Generic (PLEG): container finished" podID="c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" containerID="53cf1fb407f442abb431c7f51ae8ac8e746eef748683e72a83ecaf8640223707" exitCode=0
Oct 30 22:30:44.618754 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:44.618416    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz" event={"ID":"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd","Type":"ContainerDied","Data":"53cf1fb407f442abb431c7f51ae8ac8e746eef748683e72a83ecaf8640223707"}
Oct 30 22:30:45.624807 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:45.624774    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz" event={"ID":"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd","Type":"ContainerStarted","Data":"5140b5279af0442877e1de21dd01bea0c9cfe1a323b6785fcbb555dcd4834aa7"}
Oct 30 22:30:45.625248 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:45.624836    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz" event={"ID":"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd","Type":"ContainerStarted","Data":"84f8183963e9a5a20564a8107956358debcea2786c629cd22de632b8450f85aa"}
Oct 30 22:30:45.625248 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:45.625039    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:45.806542 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:45.806470    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz" podStartSLOduration=2.731358898 podStartE2EDuration="6.806450733s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.075091835s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:45.79565962 +0000 UTC m=+1343.795007185" watchObservedRunningTime="2026-10-30 22:30:45.806450733 +0000 UTC m=+1343.805798299"
Oct 30 22:30:46.629239 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:46.629199    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:52.639819 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:30:52.639790    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:30:59.048734 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:30:59.048697    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:31:22.699944 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:22.699863    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:31:29.048434 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:31:29.048386    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:31:29.534870 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:29.534788    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"]
Oct 30 22:31:29.535085 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:29.535055    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz" podUID="c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" containerName="kserve-container" containerID="cri-o://84f8183963e9a5a20564a8107956358debcea2786c629cd22de632b8450f85aa" gracePeriod=30
Oct 30 22:31:29.535259 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:29.535105    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz" podUID="c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" containerName="kube-rbac-proxy" containerID="cri-o://5140b5279af0442877e1de21dd01bea0c9cfe1a323b6785fcbb555dcd4834aa7" gracePeriod=30
Oct 30 22:31:29.805857 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:29.805824    4947 generic.go:356] "Generic (PLEG): container finished" podID="c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" containerID="5140b5279af0442877e1de21dd01bea0c9cfe1a323b6785fcbb555dcd4834aa7" exitCode=0
Oct 30 22:31:29.806037 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:29.805900    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz" event={"ID":"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd","Type":"ContainerDied","Data":"5140b5279af0442877e1de21dd01bea0c9cfe1a323b6785fcbb555dcd4834aa7"}
Oct 30 22:31:30.285316 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.285292    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:31:30.376536 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.376464    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kserve-provision-location\") pod \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") "
Oct 30 22:31:30.376536 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.376501    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kube-api-access-lnxsc\" (UniqueName: \"kubernetes.io/projected/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kube-api-access-lnxsc\") pod \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") "
Oct 30 22:31:30.376757 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.376567    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kube-rbac-proxy-sar-config\") pod \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") "
Oct 30 22:31:30.376757 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.376588    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-cabundle-cert\") pod \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") "
Oct 30 22:31:30.376757 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.376630    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-proxy-tls\") pod \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\" (UID: \"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd\") "
Oct 30 22:31:30.376904 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.376848    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kserve-provision-location" pod "c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" (UID: "c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:30.376972 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.376942    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kube-rbac-proxy-sar-config" pod "c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" (UID: "c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:30.377009 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.376976    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-cabundle-cert" pod "c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" (UID: "c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:30.378745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.378723    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kube-api-access-lnxsc" pod "c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" (UID: "c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd"). InnerVolumeSpecName "kube-api-access-lnxsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:30.378856 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.378764    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-proxy-tls" pod "c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" (UID: "c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:30.477743 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.477711    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.477743 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.477737    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.477743 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.477747    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.477948 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.477756    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.477948 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.477765    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnxsc\" (UniqueName: \"kubernetes.io/projected/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd-kube-api-access-lnxsc\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.812709 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.812672    4947 generic.go:356] "Generic (PLEG): container finished" podID="c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" containerID="84f8183963e9a5a20564a8107956358debcea2786c629cd22de632b8450f85aa" exitCode=0
Oct 30 22:31:30.812899 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.812745    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"
Oct 30 22:31:30.812899 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.812741    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz" event={"ID":"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd","Type":"ContainerDied","Data":"84f8183963e9a5a20564a8107956358debcea2786c629cd22de632b8450f85aa"}
Oct 30 22:31:30.812899 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.812877    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz" event={"ID":"c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd","Type":"ContainerDied","Data":"b2b34ccc79c9d3f1289c40dae038d192a4e3e42040f3a1067ccc33164b5a31b5"}
Oct 30 22:31:30.813056 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.812905    4947 scope.go:122] "RemoveContainer" containerID="5140b5279af0442877e1de21dd01bea0c9cfe1a323b6785fcbb555dcd4834aa7"
Oct 30 22:31:30.821459 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.821440    4947 scope.go:122] "RemoveContainer" containerID="84f8183963e9a5a20564a8107956358debcea2786c629cd22de632b8450f85aa"
Oct 30 22:31:30.828710 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.828692    4947 scope.go:122] "RemoveContainer" containerID="53cf1fb407f442abb431c7f51ae8ac8e746eef748683e72a83ecaf8640223707"
Oct 30 22:31:30.835863 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.835848    4947 scope.go:122] "RemoveContainer" containerID="5140b5279af0442877e1de21dd01bea0c9cfe1a323b6785fcbb555dcd4834aa7"
Oct 30 22:31:30.836108 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:31:30.836089    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5140b5279af0442877e1de21dd01bea0c9cfe1a323b6785fcbb555dcd4834aa7\": container with ID starting with 5140b5279af0442877e1de21dd01bea0c9cfe1a323b6785fcbb555dcd4834aa7 not found: ID does not exist" containerID="5140b5279af0442877e1de21dd01bea0c9cfe1a323b6785fcbb555dcd4834aa7"
Oct 30 22:31:30.836155 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.836121    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5140b5279af0442877e1de21dd01bea0c9cfe1a323b6785fcbb555dcd4834aa7"} err="failed to get container status \"5140b5279af0442877e1de21dd01bea0c9cfe1a323b6785fcbb555dcd4834aa7\": rpc error: code = NotFound desc = could not find container \"5140b5279af0442877e1de21dd01bea0c9cfe1a323b6785fcbb555dcd4834aa7\": container with ID starting with 5140b5279af0442877e1de21dd01bea0c9cfe1a323b6785fcbb555dcd4834aa7 not found: ID does not exist"
Oct 30 22:31:30.836155 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.836138    4947 scope.go:122] "RemoveContainer" containerID="84f8183963e9a5a20564a8107956358debcea2786c629cd22de632b8450f85aa"
Oct 30 22:31:30.836378 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:31:30.836359    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84f8183963e9a5a20564a8107956358debcea2786c629cd22de632b8450f85aa\": container with ID starting with 84f8183963e9a5a20564a8107956358debcea2786c629cd22de632b8450f85aa not found: ID does not exist" containerID="84f8183963e9a5a20564a8107956358debcea2786c629cd22de632b8450f85aa"
Oct 30 22:31:30.836454 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.836393    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f8183963e9a5a20564a8107956358debcea2786c629cd22de632b8450f85aa"} err="failed to get container status \"84f8183963e9a5a20564a8107956358debcea2786c629cd22de632b8450f85aa\": rpc error: code = NotFound desc = could not find container \"84f8183963e9a5a20564a8107956358debcea2786c629cd22de632b8450f85aa\": container with ID starting with 84f8183963e9a5a20564a8107956358debcea2786c629cd22de632b8450f85aa not found: ID does not exist"
Oct 30 22:31:30.836454 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.836412    4947 scope.go:122] "RemoveContainer" containerID="53cf1fb407f442abb431c7f51ae8ac8e746eef748683e72a83ecaf8640223707"
Oct 30 22:31:30.836628 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:31:30.836607    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53cf1fb407f442abb431c7f51ae8ac8e746eef748683e72a83ecaf8640223707\": container with ID starting with 53cf1fb407f442abb431c7f51ae8ac8e746eef748683e72a83ecaf8640223707 not found: ID does not exist" containerID="53cf1fb407f442abb431c7f51ae8ac8e746eef748683e72a83ecaf8640223707"
Oct 30 22:31:30.836686 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.836632    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53cf1fb407f442abb431c7f51ae8ac8e746eef748683e72a83ecaf8640223707"} err="failed to get container status \"53cf1fb407f442abb431c7f51ae8ac8e746eef748683e72a83ecaf8640223707\": rpc error: code = NotFound desc = could not find container \"53cf1fb407f442abb431c7f51ae8ac8e746eef748683e72a83ecaf8640223707\": container with ID starting with 53cf1fb407f442abb431c7f51ae8ac8e746eef748683e72a83ecaf8640223707 not found: ID does not exist"
Oct 30 22:31:30.980487 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:30.980452    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"]
Oct 30 22:31:31.030174 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:31.030127    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-predictive-lightgbm-v2-predictor-6c4bdb5675-slrkz"]
Oct 30 22:31:32.157916 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:32.157815    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" path="/var/lib/kubelet/pods/c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd/volumes"
Oct 30 22:31:49.867723 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.867691    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"]
Oct 30 22:31:49.868152 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.868039    4947 cpu_manager.go:414] "containerMap: removing container" podUID="c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" containerName="storage-initializer"
Oct 30 22:31:49.868152 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.868062    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" containerName="storage-initializer"
Oct 30 22:31:49.868152 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.868073    4947 cpu_manager.go:414] "containerMap: removing container" podUID="c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" containerName="kube-rbac-proxy"
Oct 30 22:31:49.868152 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.868079    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" containerName="kube-rbac-proxy"
Oct 30 22:31:49.868152 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.868086    4947 cpu_manager.go:414] "containerMap: removing container" podUID="c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" containerName="kserve-container"
Oct 30 22:31:49.868152 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.868091    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" containerName="kserve-container"
Oct 30 22:31:49.868375 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.868172    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" containerName="kube-rbac-proxy"
Oct 30 22:31:49.868375 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.868180    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2a33f9a-8b8f-4de7-8d73-95e6097bbcdd" containerName="kserve-container"
Oct 30 22:31:49.871176 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.871159    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:49.925859 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.925830    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:31:49.935200 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.935176    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeb50282-bf28-411c-b68d-331c4e419b5d-proxy-tls\") pod \"isvc-sklearn-predictor-54fbdb48f9-ph6zl\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:49.935278 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.935229    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeb50282-bf28-411c-b68d-331c4e419b5d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-54fbdb48f9-ph6zl\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:49.935278 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.935251    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aeb50282-bf28-411c-b68d-331c4e419b5d-cabundle-cert\") pod \"isvc-sklearn-predictor-54fbdb48f9-ph6zl\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:49.935366 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.935348    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aeb50282-bf28-411c-b68d-331c4e419b5d-kserve-provision-location\") pod \"isvc-sklearn-predictor-54fbdb48f9-ph6zl\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:49.935410 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.935397    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmrwp\" (UniqueName: \"kubernetes.io/projected/aeb50282-bf28-411c-b68d-331c4e419b5d-kube-api-access-qmrwp\") pod \"isvc-sklearn-predictor-54fbdb48f9-ph6zl\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:49.975845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.975822    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:31:49.975845 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.975822    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-sklearn-predictor-serving-cert\""
Oct 30 22:31:49.976049 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.975970    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:31:49.976049 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.975973    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:31:49.976049 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.975998    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Oct 30 22:31:49.976205 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.976057    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:31:49.977253 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:49.977232    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"]
Oct 30 22:31:50.036486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:50.036457    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeb50282-bf28-411c-b68d-331c4e419b5d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-54fbdb48f9-ph6zl\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:50.036486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:50.036488    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aeb50282-bf28-411c-b68d-331c4e419b5d-cabundle-cert\") pod \"isvc-sklearn-predictor-54fbdb48f9-ph6zl\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:50.036735 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:50.036519    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aeb50282-bf28-411c-b68d-331c4e419b5d-kserve-provision-location\") pod \"isvc-sklearn-predictor-54fbdb48f9-ph6zl\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:50.036735 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:50.036654    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmrwp\" (UniqueName: \"kubernetes.io/projected/aeb50282-bf28-411c-b68d-331c4e419b5d-kube-api-access-qmrwp\") pod \"isvc-sklearn-predictor-54fbdb48f9-ph6zl\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:50.036846 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:50.036765    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeb50282-bf28-411c-b68d-331c4e419b5d-proxy-tls\") pod \"isvc-sklearn-predictor-54fbdb48f9-ph6zl\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:50.036918 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:50.036900    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aeb50282-bf28-411c-b68d-331c4e419b5d-kserve-provision-location\") pod \"isvc-sklearn-predictor-54fbdb48f9-ph6zl\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:50.037265 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:50.037246    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aeb50282-bf28-411c-b68d-331c4e419b5d-cabundle-cert\") pod \"isvc-sklearn-predictor-54fbdb48f9-ph6zl\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:50.037349 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:50.037316    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeb50282-bf28-411c-b68d-331c4e419b5d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-54fbdb48f9-ph6zl\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:50.039373 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:50.039327    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeb50282-bf28-411c-b68d-331c4e419b5d-proxy-tls\") pod \"isvc-sklearn-predictor-54fbdb48f9-ph6zl\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:50.099672 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:50.099630    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmrwp\" (UniqueName: \"kubernetes.io/projected/aeb50282-bf28-411c-b68d-331c4e419b5d-kube-api-access-qmrwp\") pod \"isvc-sklearn-predictor-54fbdb48f9-ph6zl\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:50.187535 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:50.187453    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:50.611453 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:50.611402    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"]
Oct 30 22:31:50.614123 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:31:50.614094    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb50282_bf28_411c_b68d_331c4e419b5d.slice/crio-edad62975a2947441ce5274f10d97587489c5e3d2b03bf7796b15fc0247f6425 WatchSource:0}: Error finding container edad62975a2947441ce5274f10d97587489c5e3d2b03bf7796b15fc0247f6425: Status 404 returned error can't find the container with id edad62975a2947441ce5274f10d97587489c5e3d2b03bf7796b15fc0247f6425
Oct 30 22:31:50.895180 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:50.895146    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" event={"ID":"aeb50282-bf28-411c-b68d-331c4e419b5d","Type":"ContainerStarted","Data":"cb8623ca0465600e4bdf820f65a1f33e97435697891d2275c822407da8e791f0"}
Oct 30 22:31:50.895667 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:50.895201    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" event={"ID":"aeb50282-bf28-411c-b68d-331c4e419b5d","Type":"ContainerStarted","Data":"edad62975a2947441ce5274f10d97587489c5e3d2b03bf7796b15fc0247f6425"}
Oct 30 22:31:54.913047 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:54.913012    4947 generic.go:356] "Generic (PLEG): container finished" podID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerID="cb8623ca0465600e4bdf820f65a1f33e97435697891d2275c822407da8e791f0" exitCode=0
Oct 30 22:31:54.913474 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:54.913070    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" event={"ID":"aeb50282-bf28-411c-b68d-331c4e419b5d","Type":"ContainerDied","Data":"cb8623ca0465600e4bdf820f65a1f33e97435697891d2275c822407da8e791f0"}
Oct 30 22:31:55.919922 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:55.919883    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" event={"ID":"aeb50282-bf28-411c-b68d-331c4e419b5d","Type":"ContainerStarted","Data":"b42a04aa67934238a0dc855380db0a23fdf4e77558a54367637e7c0cd7744d18"}
Oct 30 22:31:55.920306 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:55.919975    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" event={"ID":"aeb50282-bf28-411c-b68d-331c4e419b5d","Type":"ContainerStarted","Data":"6e2919a1c8f18cd55d753a30e62079e57151784bc3de82135f2b602ae66d95c2"}
Oct 30 22:31:55.920306 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:55.920162    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:56.042669 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:56.042613    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" podStartSLOduration=2.935256274 podStartE2EDuration="7.042596381s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.107340107s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:56.040701471 +0000 UTC m=+1414.040049034" watchObservedRunningTime="2026-10-30 22:31:56.042596381 +0000 UTC m=+1414.041943943"
Oct 30 22:31:56.924168 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:56.924141    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:31:56.925522 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:56.925492    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Oct 30 22:31:57.928588 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:31:57.928539    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Oct 30 22:31:59.048520 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:31:59.048482    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:32:02.933783 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:32:02.933754    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:32:02.934280 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:32:02.934244    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Oct 30 22:32:12.934747 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:32:12.934707    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Oct 30 22:32:22.934322 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:32:22.934284    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Oct 30 22:32:29.049266 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:32:29.049225    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:32:32.934683 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:32:32.934643    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Oct 30 22:32:42.934587 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:32:42.934544    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Oct 30 22:32:52.934354 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:32:52.934305    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Oct 30 22:32:59.048247 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:32:59.048210    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:33:02.934978 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:02.934892    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Oct 30 22:33:06.153133 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:06.153088    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Oct 30 22:33:16.158321 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:16.158290    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:33:19.422212 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:19.422184    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-sklearn-predictor-54fbdb48f9-ph6zl_aeb50282-bf28-411c-b68d-331c4e419b5d/kserve-container/0.log"
Oct 30 22:33:21.492764 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:21.492733    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-sklearn-predictor-54fbdb48f9-ph6zl_aeb50282-bf28-411c-b68d-331c4e419b5d/kserve-container/0.log"
Oct 30 22:33:22.276246 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:22.276143    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:33:22.283926 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:22.283905    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:33:23.611911 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:23.611883    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-sklearn-predictor-54fbdb48f9-ph6zl_aeb50282-bf28-411c-b68d-331c4e419b5d/kserve-container/0.log"
Oct 30 22:33:25.682161 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:25.682111    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-sklearn-predictor-54fbdb48f9-ph6zl_aeb50282-bf28-411c-b68d-331c4e419b5d/kserve-container/0.log"
Oct 30 22:33:27.752880 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:27.752843    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-sklearn-predictor-54fbdb48f9-ph6zl_aeb50282-bf28-411c-b68d-331c4e419b5d/kserve-container/0.log"
Oct 30 22:33:29.049690 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:33:29.049654    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:33:29.830301 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:29.830270    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-sklearn-predictor-54fbdb48f9-ph6zl_aeb50282-bf28-411c-b68d-331c4e419b5d/kserve-container/0.log"
Oct 30 22:33:31.900077 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:31.900012    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-sklearn-predictor-54fbdb48f9-ph6zl_aeb50282-bf28-411c-b68d-331c4e419b5d/kserve-container/0.log"
Oct 30 22:33:33.971054 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:33.971024    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-sklearn-predictor-54fbdb48f9-ph6zl_aeb50282-bf28-411c-b68d-331c4e419b5d/kserve-container/0.log"
Oct 30 22:33:36.041184 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:36.041155    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-sklearn-predictor-54fbdb48f9-ph6zl_aeb50282-bf28-411c-b68d-331c4e419b5d/kserve-container/0.log"
Oct 30 22:33:38.159063 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:38.159026    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-sklearn-predictor-54fbdb48f9-ph6zl_aeb50282-bf28-411c-b68d-331c4e419b5d/kserve-container/0.log"
Oct 30 22:33:40.245286 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:40.245247    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-sklearn-predictor-54fbdb48f9-ph6zl_aeb50282-bf28-411c-b68d-331c4e419b5d/kserve-container/0.log"
Oct 30 22:33:42.315430 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:42.315396    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-sklearn-predictor-54fbdb48f9-ph6zl_aeb50282-bf28-411c-b68d-331c4e419b5d/kserve-container/0.log"
Oct 30 22:33:44.433570 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:44.433542    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-sklearn-predictor-54fbdb48f9-ph6zl_aeb50282-bf28-411c-b68d-331c4e419b5d/kserve-container/0.log"
Oct 30 22:33:46.504147 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:46.504104    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-sklearn-predictor-54fbdb48f9-ph6zl_aeb50282-bf28-411c-b68d-331c4e419b5d/kserve-container/0.log"
Oct 30 22:33:48.574371 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:48.574316    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-sklearn-predictor-54fbdb48f9-ph6zl_aeb50282-bf28-411c-b68d-331c4e419b5d/kserve-container/0.log"
Oct 30 22:33:50.645235 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:50.645208    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-bc5112d610_isvc-sklearn-predictor-54fbdb48f9-ph6zl_aeb50282-bf28-411c-b68d-331c4e419b5d/kserve-container/0.log"
Oct 30 22:33:52.224555 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:52.224520    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"]
Oct 30 22:33:52.225028 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:52.224880    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kserve-container" containerID="cri-o://6e2919a1c8f18cd55d753a30e62079e57151784bc3de82135f2b602ae66d95c2" gracePeriod=30
Oct 30 22:33:52.225028 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:52.224897    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kube-rbac-proxy" containerID="cri-o://b42a04aa67934238a0dc855380db0a23fdf4e77558a54367637e7c0cd7744d18" gracePeriod=30
Oct 30 22:33:52.401257 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:52.401213    4947 generic.go:356] "Generic (PLEG): container finished" podID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerID="b42a04aa67934238a0dc855380db0a23fdf4e77558a54367637e7c0cd7744d18" exitCode=0
Oct 30 22:33:52.401449 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:52.401280    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" event={"ID":"aeb50282-bf28-411c-b68d-331c4e419b5d","Type":"ContainerDied","Data":"b42a04aa67934238a0dc855380db0a23fdf4e77558a54367637e7c0cd7744d18"}
Oct 30 22:33:52.929738 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:52.929691    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Oct 30 22:33:56.153766 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:56.153725    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Oct 30 22:33:56.973670 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:56.973645    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:33:57.068980 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.068890    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeb50282-bf28-411c-b68d-331c4e419b5d-kube-api-access-qmrwp\" (UniqueName: \"kubernetes.io/projected/aeb50282-bf28-411c-b68d-331c4e419b5d-kube-api-access-qmrwp\") pod \"aeb50282-bf28-411c-b68d-331c4e419b5d\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") "
Oct 30 22:33:57.068980 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.068953    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeb50282-bf28-411c-b68d-331c4e419b5d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aeb50282-bf28-411c-b68d-331c4e419b5d-cabundle-cert\") pod \"aeb50282-bf28-411c-b68d-331c4e419b5d\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") "
Oct 30 22:33:57.069210 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.068983    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeb50282-bf28-411c-b68d-331c4e419b5d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeb50282-bf28-411c-b68d-331c4e419b5d-proxy-tls\") pod \"aeb50282-bf28-411c-b68d-331c4e419b5d\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") "
Oct 30 22:33:57.069210 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.069035    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeb50282-bf28-411c-b68d-331c4e419b5d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeb50282-bf28-411c-b68d-331c4e419b5d-kube-rbac-proxy-sar-config\") pod \"aeb50282-bf28-411c-b68d-331c4e419b5d\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") "
Oct 30 22:33:57.069210 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.069102    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeb50282-bf28-411c-b68d-331c4e419b5d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aeb50282-bf28-411c-b68d-331c4e419b5d-kserve-provision-location\") pod \"aeb50282-bf28-411c-b68d-331c4e419b5d\" (UID: \"aeb50282-bf28-411c-b68d-331c4e419b5d\") "
Oct 30 22:33:57.069468 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.069433    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb50282-bf28-411c-b68d-331c4e419b5d-cabundle-cert" pod "aeb50282-bf28-411c-b68d-331c4e419b5d" (UID: "aeb50282-bf28-411c-b68d-331c4e419b5d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:57.069587 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.069467    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb50282-bf28-411c-b68d-331c4e419b5d-kube-rbac-proxy-sar-config" pod "aeb50282-bf28-411c-b68d-331c4e419b5d" (UID: "aeb50282-bf28-411c-b68d-331c4e419b5d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:57.069587 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.069494    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb50282-bf28-411c-b68d-331c4e419b5d-kserve-provision-location" pod "aeb50282-bf28-411c-b68d-331c4e419b5d" (UID: "aeb50282-bf28-411c-b68d-331c4e419b5d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:57.071253 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.071233    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb50282-bf28-411c-b68d-331c4e419b5d-kube-api-access-qmrwp" pod "aeb50282-bf28-411c-b68d-331c4e419b5d" (UID: "aeb50282-bf28-411c-b68d-331c4e419b5d"). InnerVolumeSpecName "kube-api-access-qmrwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:57.071306 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.071260    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeb50282-bf28-411c-b68d-331c4e419b5d-proxy-tls" pod "aeb50282-bf28-411c-b68d-331c4e419b5d" (UID: "aeb50282-bf28-411c-b68d-331c4e419b5d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:57.170658 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.170625    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aeb50282-bf28-411c-b68d-331c4e419b5d-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:57.170658 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.170652    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmrwp\" (UniqueName: \"kubernetes.io/projected/aeb50282-bf28-411c-b68d-331c4e419b5d-kube-api-access-qmrwp\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:57.170658 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.170662    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aeb50282-bf28-411c-b68d-331c4e419b5d-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:57.171084 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.170671    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeb50282-bf28-411c-b68d-331c4e419b5d-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:57.171084 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.170680    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeb50282-bf28-411c-b68d-331c4e419b5d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:57.421991 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.421913    4947 generic.go:356] "Generic (PLEG): container finished" podID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerID="6e2919a1c8f18cd55d753a30e62079e57151784bc3de82135f2b602ae66d95c2" exitCode=0
Oct 30 22:33:57.421991 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.421949    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" event={"ID":"aeb50282-bf28-411c-b68d-331c4e419b5d","Type":"ContainerDied","Data":"6e2919a1c8f18cd55d753a30e62079e57151784bc3de82135f2b602ae66d95c2"}
Oct 30 22:33:57.421991 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.421977    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"
Oct 30 22:33:57.421991 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.421994    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl" event={"ID":"aeb50282-bf28-411c-b68d-331c4e419b5d","Type":"ContainerDied","Data":"edad62975a2947441ce5274f10d97587489c5e3d2b03bf7796b15fc0247f6425"}
Oct 30 22:33:57.422280 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.422022    4947 scope.go:122] "RemoveContainer" containerID="b42a04aa67934238a0dc855380db0a23fdf4e77558a54367637e7c0cd7744d18"
Oct 30 22:33:57.430781 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.430764    4947 scope.go:122] "RemoveContainer" containerID="6e2919a1c8f18cd55d753a30e62079e57151784bc3de82135f2b602ae66d95c2"
Oct 30 22:33:57.438072 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.438053    4947 scope.go:122] "RemoveContainer" containerID="cb8623ca0465600e4bdf820f65a1f33e97435697891d2275c822407da8e791f0"
Oct 30 22:33:57.445398 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.445381    4947 scope.go:122] "RemoveContainer" containerID="b42a04aa67934238a0dc855380db0a23fdf4e77558a54367637e7c0cd7744d18"
Oct 30 22:33:57.445644 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:33:57.445625    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b42a04aa67934238a0dc855380db0a23fdf4e77558a54367637e7c0cd7744d18\": container with ID starting with b42a04aa67934238a0dc855380db0a23fdf4e77558a54367637e7c0cd7744d18 not found: ID does not exist" containerID="b42a04aa67934238a0dc855380db0a23fdf4e77558a54367637e7c0cd7744d18"
Oct 30 22:33:57.445692 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.445654    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b42a04aa67934238a0dc855380db0a23fdf4e77558a54367637e7c0cd7744d18"} err="failed to get container status \"b42a04aa67934238a0dc855380db0a23fdf4e77558a54367637e7c0cd7744d18\": rpc error: code = NotFound desc = could not find container \"b42a04aa67934238a0dc855380db0a23fdf4e77558a54367637e7c0cd7744d18\": container with ID starting with b42a04aa67934238a0dc855380db0a23fdf4e77558a54367637e7c0cd7744d18 not found: ID does not exist"
Oct 30 22:33:57.445692 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.445675    4947 scope.go:122] "RemoveContainer" containerID="6e2919a1c8f18cd55d753a30e62079e57151784bc3de82135f2b602ae66d95c2"
Oct 30 22:33:57.445882 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:33:57.445866    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e2919a1c8f18cd55d753a30e62079e57151784bc3de82135f2b602ae66d95c2\": container with ID starting with 6e2919a1c8f18cd55d753a30e62079e57151784bc3de82135f2b602ae66d95c2 not found: ID does not exist" containerID="6e2919a1c8f18cd55d753a30e62079e57151784bc3de82135f2b602ae66d95c2"
Oct 30 22:33:57.445921 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.445887    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e2919a1c8f18cd55d753a30e62079e57151784bc3de82135f2b602ae66d95c2"} err="failed to get container status \"6e2919a1c8f18cd55d753a30e62079e57151784bc3de82135f2b602ae66d95c2\": rpc error: code = NotFound desc = could not find container \"6e2919a1c8f18cd55d753a30e62079e57151784bc3de82135f2b602ae66d95c2\": container with ID starting with 6e2919a1c8f18cd55d753a30e62079e57151784bc3de82135f2b602ae66d95c2 not found: ID does not exist"
Oct 30 22:33:57.445921 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.445901    4947 scope.go:122] "RemoveContainer" containerID="cb8623ca0465600e4bdf820f65a1f33e97435697891d2275c822407da8e791f0"
Oct 30 22:33:57.446127 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:33:57.446111    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8623ca0465600e4bdf820f65a1f33e97435697891d2275c822407da8e791f0\": container with ID starting with cb8623ca0465600e4bdf820f65a1f33e97435697891d2275c822407da8e791f0 not found: ID does not exist" containerID="cb8623ca0465600e4bdf820f65a1f33e97435697891d2275c822407da8e791f0"
Oct 30 22:33:57.446165 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.446131    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8623ca0465600e4bdf820f65a1f33e97435697891d2275c822407da8e791f0"} err="failed to get container status \"cb8623ca0465600e4bdf820f65a1f33e97435697891d2275c822407da8e791f0\": rpc error: code = NotFound desc = could not find container \"cb8623ca0465600e4bdf820f65a1f33e97435697891d2275c822407da8e791f0\": container with ID starting with cb8623ca0465600e4bdf820f65a1f33e97435697891d2275c822407da8e791f0 not found: ID does not exist"
Oct 30 22:33:57.589764 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.589728    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"]
Oct 30 22:33:57.598453 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:57.598427    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-predictor-54fbdb48f9-ph6zl"]
Oct 30 22:33:58.158258 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:33:58.158224    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" path="/var/lib/kubelet/pods/aeb50282-bf28-411c-b68d-331c4e419b5d/volumes"
Oct 30 22:33:59.050208 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:33:59.050173    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:34:01.232084 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.232046    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"]
Oct 30 22:34:01.232508 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.232395    4947 cpu_manager.go:414] "containerMap: removing container" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="storage-initializer"
Oct 30 22:34:01.232508 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.232415    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="storage-initializer"
Oct 30 22:34:01.232508 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.232425    4947 cpu_manager.go:414] "containerMap: removing container" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kserve-container"
Oct 30 22:34:01.232508 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.232430    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kserve-container"
Oct 30 22:34:01.232508 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.232444    4947 cpu_manager.go:414] "containerMap: removing container" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kube-rbac-proxy"
Oct 30 22:34:01.232508 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.232450    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kube-rbac-proxy"
Oct 30 22:34:01.232508 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.232497    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kube-rbac-proxy"
Oct 30 22:34:01.232508 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.232509    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeb50282-bf28-411c-b68d-331c4e419b5d" containerName="kserve-container"
Oct 30 22:34:01.237076 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.237059    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:34:01.292196 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.292164    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:34:01.292196 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.292165    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Oct 30 22:34:01.306561 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.306530    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzphx\" (UniqueName: \"kubernetes.io/projected/ea840d21-f26e-406e-9df9-4289582149b1-kube-api-access-dzphx\") pod \"sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") " pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:34:01.306681 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.306608    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea840d21-f26e-406e-9df9-4289582149b1-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") " pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:34:01.306785 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.306767    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea840d21-f26e-406e-9df9-4289582149b1-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") " pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:34:01.306859 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.306847    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea840d21-f26e-406e-9df9-4289582149b1-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") " pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:34:01.306923 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.306902    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea840d21-f26e-406e-9df9-4289582149b1-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") " pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:34:01.341520 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.341484    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:34:01.341520 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.341497    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:34:01.341520 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.341521    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:34:01.341943 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.341916    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:34:01.342066 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.341952    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 30 22:34:01.343379 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.343363    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"]
Oct 30 22:34:01.407440 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.407304    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea840d21-f26e-406e-9df9-4289582149b1-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") " pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:34:01.407440 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.407376    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea840d21-f26e-406e-9df9-4289582149b1-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") " pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:34:01.407440 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.407429    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea840d21-f26e-406e-9df9-4289582149b1-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") " pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:34:01.407811 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.407462    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea840d21-f26e-406e-9df9-4289582149b1-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") " pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:34:01.407811 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.407524    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzphx\" (UniqueName: \"kubernetes.io/projected/ea840d21-f26e-406e-9df9-4289582149b1-kube-api-access-dzphx\") pod \"sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") " pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:34:01.408130 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.408103    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea840d21-f26e-406e-9df9-4289582149b1-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") " pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:34:01.408549 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.408521    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea840d21-f26e-406e-9df9-4289582149b1-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") " pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:34:01.408643 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.408575    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea840d21-f26e-406e-9df9-4289582149b1-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") " pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:34:01.410428 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.410408    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea840d21-f26e-406e-9df9-4289582149b1-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") " pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:34:01.470563 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.470528    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzphx\" (UniqueName: \"kubernetes.io/projected/ea840d21-f26e-406e-9df9-4289582149b1-kube-api-access-dzphx\") pod \"sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") " pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:34:01.553417 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.553381    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:34:01.985402 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:01.985312    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"]
Oct 30 22:34:01.989598 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:34:01.989571    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea840d21_f26e_406e_9df9_4289582149b1.slice/crio-4e145bcf16b1e7d55ed4ff09515a8ae573c2e6fb9bb18e239c8ac2702999d253 WatchSource:0}: Error finding container 4e145bcf16b1e7d55ed4ff09515a8ae573c2e6fb9bb18e239c8ac2702999d253: Status 404 returned error can't find the container with id 4e145bcf16b1e7d55ed4ff09515a8ae573c2e6fb9bb18e239c8ac2702999d253
Oct 30 22:34:02.450609 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:02.450576    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5" event={"ID":"ea840d21-f26e-406e-9df9-4289582149b1","Type":"ContainerStarted","Data":"2b72a0e25b69d70071617fd5855234f5577ae707920905b7e39a83c5780f03e6"}
Oct 30 22:34:02.450988 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:02.450632    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5" event={"ID":"ea840d21-f26e-406e-9df9-4289582149b1","Type":"ContainerStarted","Data":"4e145bcf16b1e7d55ed4ff09515a8ae573c2e6fb9bb18e239c8ac2702999d253"}
Oct 30 22:34:06.468261 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:06.468180    4947 generic.go:356] "Generic (PLEG): container finished" podID="ea840d21-f26e-406e-9df9-4289582149b1" containerID="2b72a0e25b69d70071617fd5855234f5577ae707920905b7e39a83c5780f03e6" exitCode=0
Oct 30 22:34:06.468261 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:06.468249    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5" event={"ID":"ea840d21-f26e-406e-9df9-4289582149b1","Type":"ContainerDied","Data":"2b72a0e25b69d70071617fd5855234f5577ae707920905b7e39a83c5780f03e6"}
Oct 30 22:34:06.469430 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:34:06.469411    4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:34:29.054704 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:34:29.054667    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:34:59.051733 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:34:59.051671    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:35:29.050798 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:35:29.050758    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:35:31.537540 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.537500    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"]
Oct 30 22:35:31.718906 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.718867    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"]
Oct 30 22:35:31.719089 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.718988    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:31.773615 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.773585    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"openshift-service-ca.crt\""
Oct 30 22:35:31.773782 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.773638    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"default-dockercfg-s9cqw\""
Oct 30 22:35:31.813310 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.813228    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnkt9\" (UniqueName: \"kubernetes.io/projected/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kube-api-access-xnkt9\") pod \"isvc-logger-predictor-7d97d49c7-vjjsm\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") " pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:31.813310 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.813282    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-proxy-tls\") pod \"isvc-logger-predictor-7d97d49c7-vjjsm\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") " pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:31.813537 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.813315    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7d97d49c7-vjjsm\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") " pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:31.813537 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.813462    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-cabundle-cert\") pod \"isvc-logger-predictor-7d97d49c7-vjjsm\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") " pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:31.813537 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.813523    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kserve-provision-location\") pod \"isvc-logger-predictor-7d97d49c7-vjjsm\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") " pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:31.822718 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.822692    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Oct 30 22:35:31.822718 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.822693    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"kube-root-ca.crt\""
Oct 30 22:35:31.822946 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.822693    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-logger-predictor-serving-cert\""
Oct 30 22:35:31.822946 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.822730    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:35:31.822946 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.822750    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"seaweedfs-s3-creds\""
Oct 30 22:35:31.914284 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.914012    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnkt9\" (UniqueName: \"kubernetes.io/projected/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kube-api-access-xnkt9\") pod \"isvc-logger-predictor-7d97d49c7-vjjsm\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") " pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:31.914284 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.914062    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-proxy-tls\") pod \"isvc-logger-predictor-7d97d49c7-vjjsm\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") " pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:31.914284 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.914098    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7d97d49c7-vjjsm\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") " pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:31.914284 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.914125    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-cabundle-cert\") pod \"isvc-logger-predictor-7d97d49c7-vjjsm\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") " pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:31.914284 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.914172    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kserve-provision-location\") pod \"isvc-logger-predictor-7d97d49c7-vjjsm\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") " pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:31.914724 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.914703    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kserve-provision-location\") pod \"isvc-logger-predictor-7d97d49c7-vjjsm\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") " pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:31.915012 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.914995    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7d97d49c7-vjjsm\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") " pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:31.915344 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.915314    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-cabundle-cert\") pod \"isvc-logger-predictor-7d97d49c7-vjjsm\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") " pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:31.917201 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.917179    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-proxy-tls\") pod \"isvc-logger-predictor-7d97d49c7-vjjsm\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") " pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:31.998988 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:31.998951    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnkt9\" (UniqueName: \"kubernetes.io/projected/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kube-api-access-xnkt9\") pod \"isvc-logger-predictor-7d97d49c7-vjjsm\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") " pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:32.248965 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:32.248879    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:32.784674 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:32.784643    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"]
Oct 30 22:35:32.789947 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:35:32.789913    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc9cad0_d37e_4e0c_aa29_2e5d72f1b7be.slice/crio-ccde8d246522d30b0ba68b882d719aafe5a1226b14cc2a86c42f9160ad8f5a1d WatchSource:0}: Error finding container ccde8d246522d30b0ba68b882d719aafe5a1226b14cc2a86c42f9160ad8f5a1d: Status 404 returned error can't find the container with id ccde8d246522d30b0ba68b882d719aafe5a1226b14cc2a86c42f9160ad8f5a1d
Oct 30 22:35:32.881796 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:32.881757    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" event={"ID":"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be","Type":"ContainerStarted","Data":"ccde8d246522d30b0ba68b882d719aafe5a1226b14cc2a86c42f9160ad8f5a1d"}
Oct 30 22:35:33.889569 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:33.889524    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" event={"ID":"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be","Type":"ContainerStarted","Data":"37ba02cdc6fa340adfcdad6b5de6d206f00583a34c1a345c9ec21d4dad5fb721"}
Oct 30 22:35:37.911313 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:37.911274    4947 generic.go:356] "Generic (PLEG): container finished" podID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerID="37ba02cdc6fa340adfcdad6b5de6d206f00583a34c1a345c9ec21d4dad5fb721" exitCode=0
Oct 30 22:35:37.911313 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:37.911317    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" event={"ID":"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be","Type":"ContainerDied","Data":"37ba02cdc6fa340adfcdad6b5de6d206f00583a34c1a345c9ec21d4dad5fb721"}
Oct 30 22:35:38.919559 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:38.919527    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" event={"ID":"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be","Type":"ContainerStarted","Data":"1017dd1128093e225b14a1407932732374eaa903068ef31ef7baad76fa2b6ed7"}
Oct 30 22:35:38.919994 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:38.919607    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" event={"ID":"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be","Type":"ContainerStarted","Data":"1520a77f1ea57d0826662c194a11f371942f253c0fce67960d40ecb7f6cb5a4e"}
Oct 30 22:35:38.919994 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:38.919637    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" event={"ID":"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be","Type":"ContainerStarted","Data":"0da7210e335763e59470809052103873905658e3a28a679b4ecf4b9e47789a5f"}
Oct 30 22:35:38.920473 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:38.920442    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:38.920602 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:38.920516    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:38.920602 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:38.920554    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:38.922057 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:38.921985    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Oct 30 22:35:38.922858 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:38.922825    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:35:39.043376 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:39.043293    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podStartSLOduration=3.654403506 podStartE2EDuration="8.043280988s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.388877482s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:39.042070572 +0000 UTC m=+1637.041418141" watchObservedRunningTime="2026-10-30 22:35:39.043280988 +0000 UTC m=+1637.042628603"
Oct 30 22:35:39.925592 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:39.925546    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Oct 30 22:35:39.926254 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:39.926228    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:35:44.931504 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:44.931471    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:35:44.932088 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:44.932053    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Oct 30 22:35:44.932406 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:44.932379    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:35:54.932495 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:54.932439    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Oct 30 22:35:54.932987 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:35:54.932931    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:35:59.050912 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:35:59.050867    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:36:04.932783 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:04.932734    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Oct 30 22:36:04.933278 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:04.933255    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:36:05.051886 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:05.051852    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5" event={"ID":"ea840d21-f26e-406e-9df9-4289582149b1","Type":"ContainerStarted","Data":"f78f65920e289f98b0d0dfc9b307f5267d462e9a3fa95a19e3d0b82613c82d11"}
Oct 30 22:36:05.052064 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:05.051914    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5" event={"ID":"ea840d21-f26e-406e-9df9-4289582149b1","Type":"ContainerStarted","Data":"13d03f90bb48119194d4488dfcb8343bbc1b09cf4003092ee280170b2524c3e7"}
Oct 30 22:36:05.052113 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:05.052091    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:36:05.052147 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:05.052124    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:36:05.226527 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:05.226439    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5" podStartSLOduration=2.146229209 podStartE2EDuration="2m4.226424659s" totalImagesPullingTime="1m57.964902368s" totalInitContainerRuntime="4.115293082s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:05.225521129 +0000 UTC m=+1663.224868693" watchObservedRunningTime="2026-10-30 22:36:05.226424659 +0000 UTC m=+1663.225772221"
Oct 30 22:36:11.062139 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:11.062106    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:36:14.932790 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:14.932750    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Oct 30 22:36:14.933209 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:14.933161    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:36:24.932585 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:24.932546    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Oct 30 22:36:24.933109 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:24.933085    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:36:29.047914 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:36:29.047873    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:36:34.932011 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:34.931964    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Oct 30 22:36:34.932602 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:34.932365    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:36:41.098008 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:41.097953    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5" podUID="ea840d21-f26e-406e-9df9-4289582149b1" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 30 22:36:44.932723 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:44.932668    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Oct 30 22:36:44.933167 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:44.933144    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:36:51.065213 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:51.065182    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:36:54.932819 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:54.932773    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Oct 30 22:36:54.933229 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:36:54.933151    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:36:59.048613 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:36:59.048567    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:37:01.938946 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:01.938914    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"]
Oct 30 22:37:01.939410 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:01.939138    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5" podUID="ea840d21-f26e-406e-9df9-4289582149b1" containerName="kserve-container" containerID="cri-o://13d03f90bb48119194d4488dfcb8343bbc1b09cf4003092ee280170b2524c3e7" gracePeriod=30
Oct 30 22:37:01.939410 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:01.939236    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5" podUID="ea840d21-f26e-406e-9df9-4289582149b1" containerName="kube-rbac-proxy" containerID="cri-o://f78f65920e289f98b0d0dfc9b307f5267d462e9a3fa95a19e3d0b82613c82d11" gracePeriod=30
Oct 30 22:37:02.157938 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:02.157908    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:37:02.158119 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:02.158001    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:37:02.293658 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:02.293628    4947 generic.go:356] "Generic (PLEG): container finished" podID="ea840d21-f26e-406e-9df9-4289582149b1" containerID="f78f65920e289f98b0d0dfc9b307f5267d462e9a3fa95a19e3d0b82613c82d11" exitCode=0
Oct 30 22:37:02.293847 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:02.293697    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5" event={"ID":"ea840d21-f26e-406e-9df9-4289582149b1","Type":"ContainerDied","Data":"f78f65920e289f98b0d0dfc9b307f5267d462e9a3fa95a19e3d0b82613c82d11"}
Oct 30 22:37:06.056798 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:06.056754    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5" podUID="ea840d21-f26e-406e-9df9-4289582149b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Oct 30 22:37:08.980649 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:08.980626    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:37:09.041015 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.040989    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea840d21-f26e-406e-9df9-4289582149b1-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea840d21-f26e-406e-9df9-4289582149b1-cabundle-cert\") pod \"ea840d21-f26e-406e-9df9-4289582149b1\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") "
Oct 30 22:37:09.041151 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.041037    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea840d21-f26e-406e-9df9-4289582149b1-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea840d21-f26e-406e-9df9-4289582149b1-proxy-tls\") pod \"ea840d21-f26e-406e-9df9-4289582149b1\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") "
Oct 30 22:37:09.041151 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.041071    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea840d21-f26e-406e-9df9-4289582149b1-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea840d21-f26e-406e-9df9-4289582149b1-kserve-provision-location\") pod \"ea840d21-f26e-406e-9df9-4289582149b1\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") "
Oct 30 22:37:09.041151 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.041090    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea840d21-f26e-406e-9df9-4289582149b1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea840d21-f26e-406e-9df9-4289582149b1-kube-rbac-proxy-sar-config\") pod \"ea840d21-f26e-406e-9df9-4289582149b1\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") "
Oct 30 22:37:09.041151 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.041112    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea840d21-f26e-406e-9df9-4289582149b1-kube-api-access-dzphx\" (UniqueName: \"kubernetes.io/projected/ea840d21-f26e-406e-9df9-4289582149b1-kube-api-access-dzphx\") pod \"ea840d21-f26e-406e-9df9-4289582149b1\" (UID: \"ea840d21-f26e-406e-9df9-4289582149b1\") "
Oct 30 22:37:09.041455 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.041418    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea840d21-f26e-406e-9df9-4289582149b1-kserve-provision-location" pod "ea840d21-f26e-406e-9df9-4289582149b1" (UID: "ea840d21-f26e-406e-9df9-4289582149b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:09.041455 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.041437    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea840d21-f26e-406e-9df9-4289582149b1-cabundle-cert" pod "ea840d21-f26e-406e-9df9-4289582149b1" (UID: "ea840d21-f26e-406e-9df9-4289582149b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:09.041592 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.041527    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea840d21-f26e-406e-9df9-4289582149b1-kube-rbac-proxy-sar-config" pod "ea840d21-f26e-406e-9df9-4289582149b1" (UID: "ea840d21-f26e-406e-9df9-4289582149b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:09.043361 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.043309    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea840d21-f26e-406e-9df9-4289582149b1-kube-api-access-dzphx" pod "ea840d21-f26e-406e-9df9-4289582149b1" (UID: "ea840d21-f26e-406e-9df9-4289582149b1"). InnerVolumeSpecName "kube-api-access-dzphx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:09.043527 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.043509    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea840d21-f26e-406e-9df9-4289582149b1-proxy-tls" pod "ea840d21-f26e-406e-9df9-4289582149b1" (UID: "ea840d21-f26e-406e-9df9-4289582149b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:37:09.141909 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.141857    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea840d21-f26e-406e-9df9-4289582149b1-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:09.141909 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.141877    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea840d21-f26e-406e-9df9-4289582149b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:09.141909 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.141888    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzphx\" (UniqueName: \"kubernetes.io/projected/ea840d21-f26e-406e-9df9-4289582149b1-kube-api-access-dzphx\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:09.141909 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.141898    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea840d21-f26e-406e-9df9-4289582149b1-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:09.141909 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.141906    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea840d21-f26e-406e-9df9-4289582149b1-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:09.328199 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.328169    4947 generic.go:356] "Generic (PLEG): container finished" podID="ea840d21-f26e-406e-9df9-4289582149b1" containerID="13d03f90bb48119194d4488dfcb8343bbc1b09cf4003092ee280170b2524c3e7" exitCode=0
Oct 30 22:37:09.328363 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.328226    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"
Oct 30 22:37:09.328363 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.328254    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5" event={"ID":"ea840d21-f26e-406e-9df9-4289582149b1","Type":"ContainerDied","Data":"13d03f90bb48119194d4488dfcb8343bbc1b09cf4003092ee280170b2524c3e7"}
Oct 30 22:37:09.328363 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.328314    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5" event={"ID":"ea840d21-f26e-406e-9df9-4289582149b1","Type":"ContainerDied","Data":"4e145bcf16b1e7d55ed4ff09515a8ae573c2e6fb9bb18e239c8ac2702999d253"}
Oct 30 22:37:09.328363 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.328358    4947 scope.go:122] "RemoveContainer" containerID="f78f65920e289f98b0d0dfc9b307f5267d462e9a3fa95a19e3d0b82613c82d11"
Oct 30 22:37:09.336860 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.336845    4947 scope.go:122] "RemoveContainer" containerID="13d03f90bb48119194d4488dfcb8343bbc1b09cf4003092ee280170b2524c3e7"
Oct 30 22:37:09.344161 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.344138    4947 scope.go:122] "RemoveContainer" containerID="2b72a0e25b69d70071617fd5855234f5577ae707920905b7e39a83c5780f03e6"
Oct 30 22:37:09.351317 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.351298    4947 scope.go:122] "RemoveContainer" containerID="f78f65920e289f98b0d0dfc9b307f5267d462e9a3fa95a19e3d0b82613c82d11"
Oct 30 22:37:09.351585 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:37:09.351568    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f78f65920e289f98b0d0dfc9b307f5267d462e9a3fa95a19e3d0b82613c82d11\": container with ID starting with f78f65920e289f98b0d0dfc9b307f5267d462e9a3fa95a19e3d0b82613c82d11 not found: ID does not exist" containerID="f78f65920e289f98b0d0dfc9b307f5267d462e9a3fa95a19e3d0b82613c82d11"
Oct 30 22:37:09.351636 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.351594    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78f65920e289f98b0d0dfc9b307f5267d462e9a3fa95a19e3d0b82613c82d11"} err="failed to get container status \"f78f65920e289f98b0d0dfc9b307f5267d462e9a3fa95a19e3d0b82613c82d11\": rpc error: code = NotFound desc = could not find container \"f78f65920e289f98b0d0dfc9b307f5267d462e9a3fa95a19e3d0b82613c82d11\": container with ID starting with f78f65920e289f98b0d0dfc9b307f5267d462e9a3fa95a19e3d0b82613c82d11 not found: ID does not exist"
Oct 30 22:37:09.351636 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.351614    4947 scope.go:122] "RemoveContainer" containerID="13d03f90bb48119194d4488dfcb8343bbc1b09cf4003092ee280170b2524c3e7"
Oct 30 22:37:09.351836 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:37:09.351815    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13d03f90bb48119194d4488dfcb8343bbc1b09cf4003092ee280170b2524c3e7\": container with ID starting with 13d03f90bb48119194d4488dfcb8343bbc1b09cf4003092ee280170b2524c3e7 not found: ID does not exist" containerID="13d03f90bb48119194d4488dfcb8343bbc1b09cf4003092ee280170b2524c3e7"
Oct 30 22:37:09.351902 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.351846    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13d03f90bb48119194d4488dfcb8343bbc1b09cf4003092ee280170b2524c3e7"} err="failed to get container status \"13d03f90bb48119194d4488dfcb8343bbc1b09cf4003092ee280170b2524c3e7\": rpc error: code = NotFound desc = could not find container \"13d03f90bb48119194d4488dfcb8343bbc1b09cf4003092ee280170b2524c3e7\": container with ID starting with 13d03f90bb48119194d4488dfcb8343bbc1b09cf4003092ee280170b2524c3e7 not found: ID does not exist"
Oct 30 22:37:09.351902 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.351866    4947 scope.go:122] "RemoveContainer" containerID="2b72a0e25b69d70071617fd5855234f5577ae707920905b7e39a83c5780f03e6"
Oct 30 22:37:09.352101 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:37:09.352086    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b72a0e25b69d70071617fd5855234f5577ae707920905b7e39a83c5780f03e6\": container with ID starting with 2b72a0e25b69d70071617fd5855234f5577ae707920905b7e39a83c5780f03e6 not found: ID does not exist" containerID="2b72a0e25b69d70071617fd5855234f5577ae707920905b7e39a83c5780f03e6"
Oct 30 22:37:09.352141 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.352105    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b72a0e25b69d70071617fd5855234f5577ae707920905b7e39a83c5780f03e6"} err="failed to get container status \"2b72a0e25b69d70071617fd5855234f5577ae707920905b7e39a83c5780f03e6\": rpc error: code = NotFound desc = could not find container \"2b72a0e25b69d70071617fd5855234f5577ae707920905b7e39a83c5780f03e6\": container with ID starting with 2b72a0e25b69d70071617fd5855234f5577ae707920905b7e39a83c5780f03e6 not found: ID does not exist"
Oct 30 22:37:09.496595 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.496523    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"]
Oct 30 22:37:09.502264 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:09.502237    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/sklearn-v2-mlserver-predictor-f5f84c7dc-cbbb5"]
Oct 30 22:37:10.157606 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:10.157575    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea840d21-f26e-406e-9df9-4289582149b1" path="/var/lib/kubelet/pods/ea840d21-f26e-406e-9df9-4289582149b1/volumes"
Oct 30 22:37:13.020040 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.020007    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"]
Oct 30 22:37:13.020445 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.020386    4947 cpu_manager.go:414] "containerMap: removing container" podUID="ea840d21-f26e-406e-9df9-4289582149b1" containerName="kserve-container"
Oct 30 22:37:13.020445 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.020399    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea840d21-f26e-406e-9df9-4289582149b1" containerName="kserve-container"
Oct 30 22:37:13.020445 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.020424    4947 cpu_manager.go:414] "containerMap: removing container" podUID="ea840d21-f26e-406e-9df9-4289582149b1" containerName="kube-rbac-proxy"
Oct 30 22:37:13.020445 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.020429    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea840d21-f26e-406e-9df9-4289582149b1" containerName="kube-rbac-proxy"
Oct 30 22:37:13.020445 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.020437    4947 cpu_manager.go:414] "containerMap: removing container" podUID="ea840d21-f26e-406e-9df9-4289582149b1" containerName="storage-initializer"
Oct 30 22:37:13.020445 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.020442    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea840d21-f26e-406e-9df9-4289582149b1" containerName="storage-initializer"
Oct 30 22:37:13.020647 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.020502    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea840d21-f26e-406e-9df9-4289582149b1" containerName="kserve-container"
Oct 30 22:37:13.020647 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.020512    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea840d21-f26e-406e-9df9-4289582149b1" containerName="kube-rbac-proxy"
Oct 30 22:37:13.023490 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.023474    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:13.069278 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.069243    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jslfw\" (UniqueName: \"kubernetes.io/projected/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kube-api-access-jslfw\") pod \"isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:13.069278 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.069281    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:13.069483 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.069416    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:13.069483 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.069465    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06ed568d-7069-4ac3-8f59-0d4484fd76cc-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:13.069591 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.069534    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06ed568d-7069-4ac3-8f59-0d4484fd76cc-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:13.127076 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.127041    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:37:13.127076 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.127041    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Oct 30 22:37:13.127076 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.127047    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:37:13.127279 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.127087    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:37:13.127279 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.127053    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Oct 30 22:37:13.127279 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.127058    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:37:13.127279 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.127046    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:37:13.130501 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.130466    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"]
Oct 30 22:37:13.170123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.170088    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jslfw\" (UniqueName: \"kubernetes.io/projected/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kube-api-access-jslfw\") pod \"isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:13.170284 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.170131    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:13.170284 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.170164    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:13.170284 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.170199    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06ed568d-7069-4ac3-8f59-0d4484fd76cc-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:13.170284 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.170230    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06ed568d-7069-4ac3-8f59-0d4484fd76cc-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:13.170699 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.170671    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:13.170931 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.170905    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06ed568d-7069-4ac3-8f59-0d4484fd76cc-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:13.171012 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.170928    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:13.172919 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.172896    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06ed568d-7069-4ac3-8f59-0d4484fd76cc-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:13.240382 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.240352    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jslfw\" (UniqueName: \"kubernetes.io/projected/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kube-api-access-jslfw\") pod \"isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:13.336650 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.336615    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:13.760796 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:13.760723    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"]
Oct 30 22:37:13.764196 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:37:13.764171    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ed568d_7069_4ac3_8f59_0d4484fd76cc.slice/crio-ba556301c1fb3ffe09667cfaa68bed167d6ca85e09e234b6f9d76919e204fbc0 WatchSource:0}: Error finding container ba556301c1fb3ffe09667cfaa68bed167d6ca85e09e234b6f9d76919e204fbc0: Status 404 returned error can't find the container with id ba556301c1fb3ffe09667cfaa68bed167d6ca85e09e234b6f9d76919e204fbc0
Oct 30 22:37:14.353010 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:14.352968    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz" event={"ID":"06ed568d-7069-4ac3-8f59-0d4484fd76cc","Type":"ContainerStarted","Data":"88f3f036139fe44c8224b9b55c026e3ff5ebbe76a5bc6d71ae1c4fcf3241b905"}
Oct 30 22:37:14.353440 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:14.353042    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz" event={"ID":"06ed568d-7069-4ac3-8f59-0d4484fd76cc","Type":"ContainerStarted","Data":"ba556301c1fb3ffe09667cfaa68bed167d6ca85e09e234b6f9d76919e204fbc0"}
Oct 30 22:37:17.576088 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:17.576048    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"]
Oct 30 22:37:17.576559 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:17.576316    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kserve-container" containerID="cri-o://0da7210e335763e59470809052103873905658e3a28a679b4ecf4b9e47789a5f" gracePeriod=30
Oct 30 22:37:17.576559 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:17.576377    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="agent" containerID="cri-o://1017dd1128093e225b14a1407932732374eaa903068ef31ef7baad76fa2b6ed7" gracePeriod=30
Oct 30 22:37:17.576559 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:17.576377    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kube-rbac-proxy" containerID="cri-o://1520a77f1ea57d0826662c194a11f371942f253c0fce67960d40ecb7f6cb5a4e" gracePeriod=30
Oct 30 22:37:18.370326 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:18.370293    4947 generic.go:356] "Generic (PLEG): container finished" podID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerID="1520a77f1ea57d0826662c194a11f371942f253c0fce67960d40ecb7f6cb5a4e" exitCode=0
Oct 30 22:37:18.370516 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:18.370374    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" event={"ID":"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be","Type":"ContainerDied","Data":"1520a77f1ea57d0826662c194a11f371942f253c0fce67960d40ecb7f6cb5a4e"}
Oct 30 22:37:19.926053 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:19.926014    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 30 22:37:20.379882 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:20.379852    4947 generic.go:356] "Generic (PLEG): container finished" podID="06ed568d-7069-4ac3-8f59-0d4484fd76cc" containerID="88f3f036139fe44c8224b9b55c026e3ff5ebbe76a5bc6d71ae1c4fcf3241b905" exitCode=0
Oct 30 22:37:20.380050 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:20.379925    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz" event={"ID":"06ed568d-7069-4ac3-8f59-0d4484fd76cc","Type":"ContainerDied","Data":"88f3f036139fe44c8224b9b55c026e3ff5ebbe76a5bc6d71ae1c4fcf3241b905"}
Oct 30 22:37:21.386746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:21.386715    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz" event={"ID":"06ed568d-7069-4ac3-8f59-0d4484fd76cc","Type":"ContainerStarted","Data":"c459ef2f749fa016fbe2f78ce856f2fb4fa37779797230b9c049af202cbb1a10"}
Oct 30 22:37:21.387234 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:21.386785    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz" event={"ID":"06ed568d-7069-4ac3-8f59-0d4484fd76cc","Type":"ContainerStarted","Data":"cac3b8a53be15f51983a29c41856239a75d856f336321a68eff3598cdfcbbc04"}
Oct 30 22:37:21.387234 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:21.387020    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:21.509399 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:21.509305    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz" podStartSLOduration=3.221633764 podStartE2EDuration="9.5092873s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.287653536s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:21.508316466 +0000 UTC m=+1739.507664040" watchObservedRunningTime="2026-10-30 22:37:21.5092873 +0000 UTC m=+1739.508634863"
Oct 30 22:37:22.155772 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:22.155590    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Oct 30 22:37:22.156061 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:22.155868    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:37:22.391256 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:22.391225    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:22.392770 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:22.392743    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz" podUID="06ed568d-7069-4ac3-8f59-0d4484fd76cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Oct 30 22:37:23.398241 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:23.398209    4947 generic.go:356] "Generic (PLEG): container finished" podID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerID="0da7210e335763e59470809052103873905658e3a28a679b4ecf4b9e47789a5f" exitCode=0
Oct 30 22:37:23.398653 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:23.398278    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" event={"ID":"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be","Type":"ContainerDied","Data":"0da7210e335763e59470809052103873905658e3a28a679b4ecf4b9e47789a5f"}
Oct 30 22:37:23.398653 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:23.398606    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz" podUID="06ed568d-7069-4ac3-8f59-0d4484fd76cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Oct 30 22:37:24.925939 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:24.925892    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 30 22:37:28.403490 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:28.403462    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:28.403973 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:28.403942    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz" podUID="06ed568d-7069-4ac3-8f59-0d4484fd76cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Oct 30 22:37:29.049704 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:37:29.049654    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:37:29.925549 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:29.925512    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 30 22:37:29.925993 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:29.925703    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:37:32.155237 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:32.155135    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Oct 30 22:37:32.155687 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:32.155478    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:37:34.925994 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:34.925958    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 30 22:37:38.405062 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:38.405034    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:39.926208 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:39.926168    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 30 22:37:42.155465 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:42.155418    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Oct 30 22:37:42.156124 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:42.156087    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:37:42.158662 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:42.158637    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:37:42.158788 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:42.158757    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:37:44.925705 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:44.925656    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 30 22:37:46.379159 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:46.379125    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"]
Oct 30 22:37:46.379620 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:46.379385    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz" podUID="06ed568d-7069-4ac3-8f59-0d4484fd76cc" containerName="kserve-container" containerID="cri-o://cac3b8a53be15f51983a29c41856239a75d856f336321a68eff3598cdfcbbc04" gracePeriod=30
Oct 30 22:37:46.379620 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:46.379399    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz" podUID="06ed568d-7069-4ac3-8f59-0d4484fd76cc" containerName="kube-rbac-proxy" containerID="cri-o://c459ef2f749fa016fbe2f78ce856f2fb4fa37779797230b9c049af202cbb1a10" gracePeriod=30
Oct 30 22:37:46.497832 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:46.497802    4947 generic.go:356] "Generic (PLEG): container finished" podID="06ed568d-7069-4ac3-8f59-0d4484fd76cc" containerID="c459ef2f749fa016fbe2f78ce856f2fb4fa37779797230b9c049af202cbb1a10" exitCode=0
Oct 30 22:37:46.497957 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:46.497874    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz" event={"ID":"06ed568d-7069-4ac3-8f59-0d4484fd76cc","Type":"ContainerDied","Data":"c459ef2f749fa016fbe2f78ce856f2fb4fa37779797230b9c049af202cbb1a10"}
Oct 30 22:37:47.325627 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.325604    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:47.466052 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.465974    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kube-api-access-jslfw\" (UniqueName: \"kubernetes.io/projected/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kube-api-access-jslfw\") pod \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") "
Oct 30 22:37:47.466052 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.466028    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kube-rbac-proxy-sar-config\") pod \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") "
Oct 30 22:37:47.466559 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.466066    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ed568d-7069-4ac3-8f59-0d4484fd76cc-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06ed568d-7069-4ac3-8f59-0d4484fd76cc-cabundle-cert\") pod \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") "
Oct 30 22:37:47.466559 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.466088    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06ed568d-7069-4ac3-8f59-0d4484fd76cc-proxy-tls\" (UniqueName: \"kubernetes.io/secret/06ed568d-7069-4ac3-8f59-0d4484fd76cc-proxy-tls\") pod \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") "
Oct 30 22:37:47.466559 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.466128    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kserve-provision-location\") pod \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\" (UID: \"06ed568d-7069-4ac3-8f59-0d4484fd76cc\") "
Oct 30 22:37:47.466559 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.466500    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ed568d-7069-4ac3-8f59-0d4484fd76cc-cabundle-cert" pod "06ed568d-7069-4ac3-8f59-0d4484fd76cc" (UID: "06ed568d-7069-4ac3-8f59-0d4484fd76cc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:47.466559 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.466508    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kube-rbac-proxy-sar-config" pod "06ed568d-7069-4ac3-8f59-0d4484fd76cc" (UID: "06ed568d-7069-4ac3-8f59-0d4484fd76cc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:47.468262 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.468240    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kube-api-access-jslfw" pod "06ed568d-7069-4ac3-8f59-0d4484fd76cc" (UID: "06ed568d-7069-4ac3-8f59-0d4484fd76cc"). InnerVolumeSpecName "kube-api-access-jslfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:47.468436 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.468419    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ed568d-7069-4ac3-8f59-0d4484fd76cc-proxy-tls" pod "06ed568d-7069-4ac3-8f59-0d4484fd76cc" (UID: "06ed568d-7069-4ac3-8f59-0d4484fd76cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:37:47.493166 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.493140    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kserve-provision-location" pod "06ed568d-7069-4ac3-8f59-0d4484fd76cc" (UID: "06ed568d-7069-4ac3-8f59-0d4484fd76cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:47.503353 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.503306    4947 generic.go:356] "Generic (PLEG): container finished" podID="06ed568d-7069-4ac3-8f59-0d4484fd76cc" containerID="cac3b8a53be15f51983a29c41856239a75d856f336321a68eff3598cdfcbbc04" exitCode=0
Oct 30 22:37:47.503462 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.503378    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz" event={"ID":"06ed568d-7069-4ac3-8f59-0d4484fd76cc","Type":"ContainerDied","Data":"cac3b8a53be15f51983a29c41856239a75d856f336321a68eff3598cdfcbbc04"}
Oct 30 22:37:47.503462 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.503427    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz" event={"ID":"06ed568d-7069-4ac3-8f59-0d4484fd76cc","Type":"ContainerDied","Data":"ba556301c1fb3ffe09667cfaa68bed167d6ca85e09e234b6f9d76919e204fbc0"}
Oct 30 22:37:47.503462 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.503454    4947 scope.go:122] "RemoveContainer" containerID="c459ef2f749fa016fbe2f78ce856f2fb4fa37779797230b9c049af202cbb1a10"
Oct 30 22:37:47.503590 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.503382    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"
Oct 30 22:37:47.513587 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.513568    4947 scope.go:122] "RemoveContainer" containerID="cac3b8a53be15f51983a29c41856239a75d856f336321a68eff3598cdfcbbc04"
Oct 30 22:37:47.521506 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.521491    4947 scope.go:122] "RemoveContainer" containerID="88f3f036139fe44c8224b9b55c026e3ff5ebbe76a5bc6d71ae1c4fcf3241b905"
Oct 30 22:37:47.529436 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.529416    4947 scope.go:122] "RemoveContainer" containerID="c459ef2f749fa016fbe2f78ce856f2fb4fa37779797230b9c049af202cbb1a10"
Oct 30 22:37:47.529706 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:37:47.529685    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c459ef2f749fa016fbe2f78ce856f2fb4fa37779797230b9c049af202cbb1a10\": container with ID starting with c459ef2f749fa016fbe2f78ce856f2fb4fa37779797230b9c049af202cbb1a10 not found: ID does not exist" containerID="c459ef2f749fa016fbe2f78ce856f2fb4fa37779797230b9c049af202cbb1a10"
Oct 30 22:37:47.529769 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.529716    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c459ef2f749fa016fbe2f78ce856f2fb4fa37779797230b9c049af202cbb1a10"} err="failed to get container status \"c459ef2f749fa016fbe2f78ce856f2fb4fa37779797230b9c049af202cbb1a10\": rpc error: code = NotFound desc = could not find container \"c459ef2f749fa016fbe2f78ce856f2fb4fa37779797230b9c049af202cbb1a10\": container with ID starting with c459ef2f749fa016fbe2f78ce856f2fb4fa37779797230b9c049af202cbb1a10 not found: ID does not exist"
Oct 30 22:37:47.529769 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.529732    4947 scope.go:122] "RemoveContainer" containerID="cac3b8a53be15f51983a29c41856239a75d856f336321a68eff3598cdfcbbc04"
Oct 30 22:37:47.529971 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:37:47.529954    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac3b8a53be15f51983a29c41856239a75d856f336321a68eff3598cdfcbbc04\": container with ID starting with cac3b8a53be15f51983a29c41856239a75d856f336321a68eff3598cdfcbbc04 not found: ID does not exist" containerID="cac3b8a53be15f51983a29c41856239a75d856f336321a68eff3598cdfcbbc04"
Oct 30 22:37:47.530007 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.529979    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac3b8a53be15f51983a29c41856239a75d856f336321a68eff3598cdfcbbc04"} err="failed to get container status \"cac3b8a53be15f51983a29c41856239a75d856f336321a68eff3598cdfcbbc04\": rpc error: code = NotFound desc = could not find container \"cac3b8a53be15f51983a29c41856239a75d856f336321a68eff3598cdfcbbc04\": container with ID starting with cac3b8a53be15f51983a29c41856239a75d856f336321a68eff3598cdfcbbc04 not found: ID does not exist"
Oct 30 22:37:47.530007 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.529995    4947 scope.go:122] "RemoveContainer" containerID="88f3f036139fe44c8224b9b55c026e3ff5ebbe76a5bc6d71ae1c4fcf3241b905"
Oct 30 22:37:47.530235 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:37:47.530215    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f3f036139fe44c8224b9b55c026e3ff5ebbe76a5bc6d71ae1c4fcf3241b905\": container with ID starting with 88f3f036139fe44c8224b9b55c026e3ff5ebbe76a5bc6d71ae1c4fcf3241b905 not found: ID does not exist" containerID="88f3f036139fe44c8224b9b55c026e3ff5ebbe76a5bc6d71ae1c4fcf3241b905"
Oct 30 22:37:47.530287 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.530239    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f3f036139fe44c8224b9b55c026e3ff5ebbe76a5bc6d71ae1c4fcf3241b905"} err="failed to get container status \"88f3f036139fe44c8224b9b55c026e3ff5ebbe76a5bc6d71ae1c4fcf3241b905\": rpc error: code = NotFound desc = could not find container \"88f3f036139fe44c8224b9b55c026e3ff5ebbe76a5bc6d71ae1c4fcf3241b905\": container with ID starting with 88f3f036139fe44c8224b9b55c026e3ff5ebbe76a5bc6d71ae1c4fcf3241b905 not found: ID does not exist"
Oct 30 22:37:47.567666 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.567645    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06ed568d-7069-4ac3-8f59-0d4484fd76cc-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:47.567756 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.567668    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:47.567756 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.567680    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jslfw\" (UniqueName: \"kubernetes.io/projected/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kube-api-access-jslfw\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:47.567756 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.567688    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06ed568d-7069-4ac3-8f59-0d4484fd76cc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:47.567756 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.567697    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06ed568d-7069-4ac3-8f59-0d4484fd76cc-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:47.683407 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.683376    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"]
Oct 30 22:37:47.689892 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.689864    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-runtime-predictor-56fddfd9df-8wwmz"]
Oct 30 22:37:47.770791 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.770765    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:37:47.869867 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.869833    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kube-api-access-xnkt9\" (UniqueName: \"kubernetes.io/projected/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kube-api-access-xnkt9\") pod \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") "
Oct 30 22:37:47.870017 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.869873    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kube-rbac-proxy-sar-config\") pod \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") "
Oct 30 22:37:47.870017 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.869910    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-proxy-tls\") pod \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") "
Oct 30 22:37:47.870017 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.869927    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-cabundle-cert\") pod \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") "
Oct 30 22:37:47.870017 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.869981    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kserve-provision-location\") pod \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\" (UID: \"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be\") "
Oct 30 22:37:47.870324 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.870296    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kserve-provision-location" pod "4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" (UID: "4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:47.870426 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.870358    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kube-rbac-proxy-sar-config" pod "4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" (UID: "4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:47.870426 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.870383    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-cabundle-cert" pod "4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" (UID: "4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:47.872097 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.872078    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-proxy-tls" pod "4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" (UID: "4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:37:47.872160 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.872141    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kube-api-access-xnkt9" pod "4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" (UID: "4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be"). InnerVolumeSpecName "kube-api-access-xnkt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:47.970502 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.970469    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:47.970502 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.970495    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnkt9\" (UniqueName: \"kubernetes.io/projected/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kube-api-access-xnkt9\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:47.970502 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.970505    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:47.970717 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.970515    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:47.970717 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:47.970525    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:37:48.158586 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.158554    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ed568d-7069-4ac3-8f59-0d4484fd76cc" path="/var/lib/kubelet/pods/06ed568d-7069-4ac3-8f59-0d4484fd76cc/volumes"
Oct 30 22:37:48.509305 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.509272    4947 generic.go:356] "Generic (PLEG): container finished" podID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerID="1017dd1128093e225b14a1407932732374eaa903068ef31ef7baad76fa2b6ed7" exitCode=137
Oct 30 22:37:48.509717 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.509370    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" event={"ID":"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be","Type":"ContainerDied","Data":"1017dd1128093e225b14a1407932732374eaa903068ef31ef7baad76fa2b6ed7"}
Oct 30 22:37:48.509717 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.509404    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"
Oct 30 22:37:48.509717 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.509449    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm" event={"ID":"4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be","Type":"ContainerDied","Data":"ccde8d246522d30b0ba68b882d719aafe5a1226b14cc2a86c42f9160ad8f5a1d"}
Oct 30 22:37:48.509717 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.509493    4947 scope.go:122] "RemoveContainer" containerID="1017dd1128093e225b14a1407932732374eaa903068ef31ef7baad76fa2b6ed7"
Oct 30 22:37:48.517415 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.517395    4947 scope.go:122] "RemoveContainer" containerID="1520a77f1ea57d0826662c194a11f371942f253c0fce67960d40ecb7f6cb5a4e"
Oct 30 22:37:48.524982 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.524947    4947 scope.go:122] "RemoveContainer" containerID="0da7210e335763e59470809052103873905658e3a28a679b4ecf4b9e47789a5f"
Oct 30 22:37:48.532073 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.532053    4947 scope.go:122] "RemoveContainer" containerID="37ba02cdc6fa340adfcdad6b5de6d206f00583a34c1a345c9ec21d4dad5fb721"
Oct 30 22:37:48.538966 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.538950    4947 scope.go:122] "RemoveContainer" containerID="1017dd1128093e225b14a1407932732374eaa903068ef31ef7baad76fa2b6ed7"
Oct 30 22:37:48.539218 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:37:48.539197    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1017dd1128093e225b14a1407932732374eaa903068ef31ef7baad76fa2b6ed7\": container with ID starting with 1017dd1128093e225b14a1407932732374eaa903068ef31ef7baad76fa2b6ed7 not found: ID does not exist" containerID="1017dd1128093e225b14a1407932732374eaa903068ef31ef7baad76fa2b6ed7"
Oct 30 22:37:48.539275 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.539227    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1017dd1128093e225b14a1407932732374eaa903068ef31ef7baad76fa2b6ed7"} err="failed to get container status \"1017dd1128093e225b14a1407932732374eaa903068ef31ef7baad76fa2b6ed7\": rpc error: code = NotFound desc = could not find container \"1017dd1128093e225b14a1407932732374eaa903068ef31ef7baad76fa2b6ed7\": container with ID starting with 1017dd1128093e225b14a1407932732374eaa903068ef31ef7baad76fa2b6ed7 not found: ID does not exist"
Oct 30 22:37:48.539275 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.539245    4947 scope.go:122] "RemoveContainer" containerID="1520a77f1ea57d0826662c194a11f371942f253c0fce67960d40ecb7f6cb5a4e"
Oct 30 22:37:48.539528 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:37:48.539510    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1520a77f1ea57d0826662c194a11f371942f253c0fce67960d40ecb7f6cb5a4e\": container with ID starting with 1520a77f1ea57d0826662c194a11f371942f253c0fce67960d40ecb7f6cb5a4e not found: ID does not exist" containerID="1520a77f1ea57d0826662c194a11f371942f253c0fce67960d40ecb7f6cb5a4e"
Oct 30 22:37:48.539579 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.539534    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1520a77f1ea57d0826662c194a11f371942f253c0fce67960d40ecb7f6cb5a4e"} err="failed to get container status \"1520a77f1ea57d0826662c194a11f371942f253c0fce67960d40ecb7f6cb5a4e\": rpc error: code = NotFound desc = could not find container \"1520a77f1ea57d0826662c194a11f371942f253c0fce67960d40ecb7f6cb5a4e\": container with ID starting with 1520a77f1ea57d0826662c194a11f371942f253c0fce67960d40ecb7f6cb5a4e not found: ID does not exist"
Oct 30 22:37:48.539579 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.539549    4947 scope.go:122] "RemoveContainer" containerID="0da7210e335763e59470809052103873905658e3a28a679b4ecf4b9e47789a5f"
Oct 30 22:37:48.539780 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:37:48.539764    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da7210e335763e59470809052103873905658e3a28a679b4ecf4b9e47789a5f\": container with ID starting with 0da7210e335763e59470809052103873905658e3a28a679b4ecf4b9e47789a5f not found: ID does not exist" containerID="0da7210e335763e59470809052103873905658e3a28a679b4ecf4b9e47789a5f"
Oct 30 22:37:48.539826 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.539785    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da7210e335763e59470809052103873905658e3a28a679b4ecf4b9e47789a5f"} err="failed to get container status \"0da7210e335763e59470809052103873905658e3a28a679b4ecf4b9e47789a5f\": rpc error: code = NotFound desc = could not find container \"0da7210e335763e59470809052103873905658e3a28a679b4ecf4b9e47789a5f\": container with ID starting with 0da7210e335763e59470809052103873905658e3a28a679b4ecf4b9e47789a5f not found: ID does not exist"
Oct 30 22:37:48.539826 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.539799    4947 scope.go:122] "RemoveContainer" containerID="37ba02cdc6fa340adfcdad6b5de6d206f00583a34c1a345c9ec21d4dad5fb721"
Oct 30 22:37:48.540012 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:37:48.539997    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ba02cdc6fa340adfcdad6b5de6d206f00583a34c1a345c9ec21d4dad5fb721\": container with ID starting with 37ba02cdc6fa340adfcdad6b5de6d206f00583a34c1a345c9ec21d4dad5fb721 not found: ID does not exist" containerID="37ba02cdc6fa340adfcdad6b5de6d206f00583a34c1a345c9ec21d4dad5fb721"
Oct 30 22:37:48.540056 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.540025    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ba02cdc6fa340adfcdad6b5de6d206f00583a34c1a345c9ec21d4dad5fb721"} err="failed to get container status \"37ba02cdc6fa340adfcdad6b5de6d206f00583a34c1a345c9ec21d4dad5fb721\": rpc error: code = NotFound desc = could not find container \"37ba02cdc6fa340adfcdad6b5de6d206f00583a34c1a345c9ec21d4dad5fb721\": container with ID starting with 37ba02cdc6fa340adfcdad6b5de6d206f00583a34c1a345c9ec21d4dad5fb721 not found: ID does not exist"
Oct 30 22:37:48.677929 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.677896    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"]
Oct 30 22:37:48.727727 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:48.727650    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-logger-predictor-7d97d49c7-vjjsm"]
Oct 30 22:37:50.159044 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.159011    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" path="/var/lib/kubelet/pods/4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be/volumes"
Oct 30 22:37:50.843437 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843399    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"]
Oct 30 22:37:50.843741 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843728    4947 cpu_manager.go:414] "containerMap: removing container" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kube-rbac-proxy"
Oct 30 22:37:50.843796 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843743    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kube-rbac-proxy"
Oct 30 22:37:50.843796 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843759    4947 cpu_manager.go:414] "containerMap: removing container" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="agent"
Oct 30 22:37:50.843796 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843768    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="agent"
Oct 30 22:37:50.843796 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843778    4947 cpu_manager.go:414] "containerMap: removing container" podUID="06ed568d-7069-4ac3-8f59-0d4484fd76cc" containerName="kserve-container"
Oct 30 22:37:50.843796 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843782    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ed568d-7069-4ac3-8f59-0d4484fd76cc" containerName="kserve-container"
Oct 30 22:37:50.843796 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843792    4947 cpu_manager.go:414] "containerMap: removing container" podUID="06ed568d-7069-4ac3-8f59-0d4484fd76cc" containerName="storage-initializer"
Oct 30 22:37:50.843796 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843797    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ed568d-7069-4ac3-8f59-0d4484fd76cc" containerName="storage-initializer"
Oct 30 22:37:50.843996 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843810    4947 cpu_manager.go:414] "containerMap: removing container" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kserve-container"
Oct 30 22:37:50.843996 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843815    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kserve-container"
Oct 30 22:37:50.843996 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843824    4947 cpu_manager.go:414] "containerMap: removing container" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="storage-initializer"
Oct 30 22:37:50.843996 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843828    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="storage-initializer"
Oct 30 22:37:50.843996 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843844    4947 cpu_manager.go:414] "containerMap: removing container" podUID="06ed568d-7069-4ac3-8f59-0d4484fd76cc" containerName="kube-rbac-proxy"
Oct 30 22:37:50.843996 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843851    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ed568d-7069-4ac3-8f59-0d4484fd76cc" containerName="kube-rbac-proxy"
Oct 30 22:37:50.843996 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843904    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ed568d-7069-4ac3-8f59-0d4484fd76cc" containerName="kube-rbac-proxy"
Oct 30 22:37:50.843996 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843914    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kube-rbac-proxy"
Oct 30 22:37:50.843996 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843925    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="agent"
Oct 30 22:37:50.843996 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843933    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ed568d-7069-4ac3-8f59-0d4484fd76cc" containerName="kserve-container"
Oct 30 22:37:50.843996 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.843940    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc9cad0-d37e-4e0c-aa29-2e5d72f1b7be" containerName="kserve-container"
Oct 30 22:37:50.848724 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.848708    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:37:50.952135 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.952104    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"seaweedfs-s3-creds\""
Oct 30 22:37:50.952440 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.952353    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Oct 30 22:37:50.952440 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.952193    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"default-dockercfg-s9cqw\""
Oct 30 22:37:50.952440 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.952376    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"openshift-service-ca.crt\""
Oct 30 22:37:50.952699 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.952442    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"kube-root-ca.crt\""
Oct 30 22:37:50.952699 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.952465    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Oct 30 22:37:50.952699 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.952472    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:37:50.954091 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.954068    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"]
Oct 30 22:37:50.995324 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.995300    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p85gk\" (UniqueName: \"kubernetes.io/projected/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kube-api-access-p85gk\") pod \"isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:37:50.995499 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.995359    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:37:50.995499 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.995413    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:37:50.995499 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.995451    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:37:50.995499 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:50.995486    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:37:51.096669 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:51.096588    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:37:51.096669 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:51.096639    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p85gk\" (UniqueName: \"kubernetes.io/projected/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kube-api-access-p85gk\") pod \"isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:37:51.096669 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:51.096659    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:37:51.096898 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:51.096747    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:37:51.096898 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:51.096846    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:37:51.097021 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:51.097006    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:37:51.097404 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:51.097377    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:37:51.097404 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:51.097382    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:37:51.099473 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:51.099455    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:37:51.159760 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:51.159728    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p85gk\" (UniqueName: \"kubernetes.io/projected/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kube-api-access-p85gk\") pod \"isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:37:51.165532 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:51.165502    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:37:51.592851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:51.592823    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"]
Oct 30 22:37:51.596113 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:37:51.596087    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9720ffe2_bc34_4367_b7b9_67b4a1a60a0b.slice/crio-4cb82a1a086bb950962e1ec6b481b09fe46d2707394d7eefd65b9748f7ab185d WatchSource:0}: Error finding container 4cb82a1a086bb950962e1ec6b481b09fe46d2707394d7eefd65b9748f7ab185d: Status 404 returned error can't find the container with id 4cb82a1a086bb950962e1ec6b481b09fe46d2707394d7eefd65b9748f7ab185d
Oct 30 22:37:52.539591 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:52.539551    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4" event={"ID":"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b","Type":"ContainerStarted","Data":"25a8ebf51d5c6e5d8d17975d88679568daaff635bc9d03045ce5d4630f7ff1ae"}
Oct 30 22:37:52.539953 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:52.539650    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4" event={"ID":"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b","Type":"ContainerStarted","Data":"4cb82a1a086bb950962e1ec6b481b09fe46d2707394d7eefd65b9748f7ab185d"}
Oct 30 22:37:53.084751 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.084718    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"]
Oct 30 22:37:53.088084 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.088069    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:53.142071 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.142041    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:37:53.191518 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.191494    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Oct 30 22:37:53.191518 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.191495    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:37:53.191808 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.191791    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:37:53.191873 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.191793    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:37:53.191873 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.191834    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:37:53.191873 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.191862    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 22:37:53.193233 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.193212    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"]
Oct 30 22:37:53.215776 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.215752    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:53.215883 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.215787    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45f8467b-804f-4cb8-b3c1-2bc430d9351e-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:53.215883 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.215813    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhstc\" (UniqueName: \"kubernetes.io/projected/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kube-api-access-jhstc\") pod \"isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:53.215959 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.215926    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:53.215994 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.215964    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45f8467b-804f-4cb8-b3c1-2bc430d9351e-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:53.317314 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.317272    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:53.317533 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.317321    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45f8467b-804f-4cb8-b3c1-2bc430d9351e-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:53.317533 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.317374    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhstc\" (UniqueName: \"kubernetes.io/projected/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kube-api-access-jhstc\") pod \"isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:53.317533 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.317424    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:53.317533 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.317494    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45f8467b-804f-4cb8-b3c1-2bc430d9351e-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:53.317768 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.317736    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:53.318094 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.318071    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45f8467b-804f-4cb8-b3c1-2bc430d9351e-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:53.318176 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.318132    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:53.320050 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.320029    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45f8467b-804f-4cb8-b3c1-2bc430d9351e-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:53.380919 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.380865    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhstc\" (UniqueName: \"kubernetes.io/projected/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kube-api-access-jhstc\") pod \"isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:53.404745 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.404726    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:53.835395 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:53.835365    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"]
Oct 30 22:37:53.838416 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:37:53.838389    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f8467b_804f_4cb8_b3c1_2bc430d9351e.slice/crio-6ab0d4bdbeb63db7a2daa2461c33aadab8406edadcec928347310d56f2deee3a WatchSource:0}: Error finding container 6ab0d4bdbeb63db7a2daa2461c33aadab8406edadcec928347310d56f2deee3a: Status 404 returned error can't find the container with id 6ab0d4bdbeb63db7a2daa2461c33aadab8406edadcec928347310d56f2deee3a
Oct 30 22:37:54.550258 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:54.550225    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9" event={"ID":"45f8467b-804f-4cb8-b3c1-2bc430d9351e","Type":"ContainerStarted","Data":"8a3092cf922ebdfa438067ed00b60a6d0d74b0975d6a8a90bd8a0ca215daa6b7"}
Oct 30 22:37:54.550480 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:54.550292    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9" event={"ID":"45f8467b-804f-4cb8-b3c1-2bc430d9351e","Type":"ContainerStarted","Data":"6ab0d4bdbeb63db7a2daa2461c33aadab8406edadcec928347310d56f2deee3a"}
Oct 30 22:37:57.566243 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:57.566215    4947 generic.go:356] "Generic (PLEG): container finished" podID="9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" containerID="25a8ebf51d5c6e5d8d17975d88679568daaff635bc9d03045ce5d4630f7ff1ae" exitCode=0
Oct 30 22:37:57.566589 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:57.566285    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4" event={"ID":"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b","Type":"ContainerDied","Data":"25a8ebf51d5c6e5d8d17975d88679568daaff635bc9d03045ce5d4630f7ff1ae"}
Oct 30 22:37:58.572299 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:58.572214    4947 generic.go:356] "Generic (PLEG): container finished" podID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" containerID="8a3092cf922ebdfa438067ed00b60a6d0d74b0975d6a8a90bd8a0ca215daa6b7" exitCode=0
Oct 30 22:37:58.572299 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:58.572284    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9" event={"ID":"45f8467b-804f-4cb8-b3c1-2bc430d9351e","Type":"ContainerDied","Data":"8a3092cf922ebdfa438067ed00b60a6d0d74b0975d6a8a90bd8a0ca215daa6b7"}
Oct 30 22:37:59.049066 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:37:59.049031    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:37:59.581191 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:59.581153    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9" event={"ID":"45f8467b-804f-4cb8-b3c1-2bc430d9351e","Type":"ContainerStarted","Data":"ddf9060804483b8573be3cddae0cc36fe24b9cfa8b5b124d4e90d20df8675687"}
Oct 30 22:37:59.581694 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:59.581225    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9" event={"ID":"45f8467b-804f-4cb8-b3c1-2bc430d9351e","Type":"ContainerStarted","Data":"2ca9042fa6c41a29928d5e14fdd2dbfd4b33996d393b1748a9df2bfd41372f2d"}
Oct 30 22:37:59.581694 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:59.581674    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:59.581807 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:59.581756    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:37:59.704996 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:37:59.704941    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9" podStartSLOduration=2.845626798 podStartE2EDuration="6.704927346s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.859300548s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:59.704057323 +0000 UTC m=+1777.703404887" watchObservedRunningTime="2026-10-30 22:37:59.704927346 +0000 UTC m=+1777.704274909"
Oct 30 22:38:05.593305 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:38:05.593274    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:38:22.307625 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:38:22.307593    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:38:22.392663 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:38:22.314395    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:38:29.054854 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:38:29.054813    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:38:35.598151 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:38:35.598104    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9" podUID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 30 22:38:45.595923 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:38:45.595878    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9" podUID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 30 22:38:55.596423 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:38:55.596393    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:38:59.052935 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:38:59.052891    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:39:01.531958 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:01.531923    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"]
Oct 30 22:39:01.532487 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:01.532195    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9" podUID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" containerName="kserve-container" containerID="cri-o://2ca9042fa6c41a29928d5e14fdd2dbfd4b33996d393b1748a9df2bfd41372f2d" gracePeriod=30
Oct 30 22:39:01.532487 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:01.532212    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9" podUID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" containerName="kube-rbac-proxy" containerID="cri-o://ddf9060804483b8573be3cddae0cc36fe24b9cfa8b5b124d4e90d20df8675687" gracePeriod=30
Oct 30 22:39:01.901683 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:01.901644    4947 generic.go:356] "Generic (PLEG): container finished" podID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" containerID="ddf9060804483b8573be3cddae0cc36fe24b9cfa8b5b124d4e90d20df8675687" exitCode=0
Oct 30 22:39:01.901876 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:01.901698    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9" event={"ID":"45f8467b-804f-4cb8-b3c1-2bc430d9351e","Type":"ContainerDied","Data":"ddf9060804483b8573be3cddae0cc36fe24b9cfa8b5b124d4e90d20df8675687"}
Oct 30 22:39:05.587954 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:05.587905    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9" podUID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Oct 30 22:39:05.594313 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:05.594273    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9" podUID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.47:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.132.0.47:8080: connect: connection refused"
Oct 30 22:39:09.587049 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.587014    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:39:09.691720 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.691641    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kserve-provision-location\") pod \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") "
Oct 30 22:39:09.691720 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.691700    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f8467b-804f-4cb8-b3c1-2bc430d9351e-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45f8467b-804f-4cb8-b3c1-2bc430d9351e-cabundle-cert\") pod \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") "
Oct 30 22:39:09.691957 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.691739    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kube-rbac-proxy-sar-config\") pod \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") "
Oct 30 22:39:09.691957 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.691774    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45f8467b-804f-4cb8-b3c1-2bc430d9351e-proxy-tls\" (UniqueName: \"kubernetes.io/secret/45f8467b-804f-4cb8-b3c1-2bc430d9351e-proxy-tls\") pod \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") "
Oct 30 22:39:09.691957 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.691857    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kube-api-access-jhstc\" (UniqueName: \"kubernetes.io/projected/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kube-api-access-jhstc\") pod \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\" (UID: \"45f8467b-804f-4cb8-b3c1-2bc430d9351e\") "
Oct 30 22:39:09.692122 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.692026    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kserve-provision-location" pod "45f8467b-804f-4cb8-b3c1-2bc430d9351e" (UID: "45f8467b-804f-4cb8-b3c1-2bc430d9351e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:09.692319 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.692290    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kube-rbac-proxy-sar-config" pod "45f8467b-804f-4cb8-b3c1-2bc430d9351e" (UID: "45f8467b-804f-4cb8-b3c1-2bc430d9351e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:09.692501 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.692479    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f8467b-804f-4cb8-b3c1-2bc430d9351e-cabundle-cert" pod "45f8467b-804f-4cb8-b3c1-2bc430d9351e" (UID: "45f8467b-804f-4cb8-b3c1-2bc430d9351e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:09.696033 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.696005    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45f8467b-804f-4cb8-b3c1-2bc430d9351e-proxy-tls" pod "45f8467b-804f-4cb8-b3c1-2bc430d9351e" (UID: "45f8467b-804f-4cb8-b3c1-2bc430d9351e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:09.696132 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.696063    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kube-api-access-jhstc" pod "45f8467b-804f-4cb8-b3c1-2bc430d9351e" (UID: "45f8467b-804f-4cb8-b3c1-2bc430d9351e"). InnerVolumeSpecName "kube-api-access-jhstc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:09.792740 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.792703    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhstc\" (UniqueName: \"kubernetes.io/projected/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kube-api-access-jhstc\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:09.792740 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.792741    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:09.792971 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.792755    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45f8467b-804f-4cb8-b3c1-2bc430d9351e-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:09.792971 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.792768    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45f8467b-804f-4cb8-b3c1-2bc430d9351e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:09.792971 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.792782    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45f8467b-804f-4cb8-b3c1-2bc430d9351e-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:09.940452 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.940410    4947 generic.go:356] "Generic (PLEG): container finished" podID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" containerID="2ca9042fa6c41a29928d5e14fdd2dbfd4b33996d393b1748a9df2bfd41372f2d" exitCode=0
Oct 30 22:39:09.940648 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.940489    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9" event={"ID":"45f8467b-804f-4cb8-b3c1-2bc430d9351e","Type":"ContainerDied","Data":"2ca9042fa6c41a29928d5e14fdd2dbfd4b33996d393b1748a9df2bfd41372f2d"}
Oct 30 22:39:09.940648 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.940568    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9" event={"ID":"45f8467b-804f-4cb8-b3c1-2bc430d9351e","Type":"ContainerDied","Data":"6ab0d4bdbeb63db7a2daa2461c33aadab8406edadcec928347310d56f2deee3a"}
Oct 30 22:39:09.940648 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.940498    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"
Oct 30 22:39:09.940648 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.940603    4947 scope.go:122] "RemoveContainer" containerID="ddf9060804483b8573be3cddae0cc36fe24b9cfa8b5b124d4e90d20df8675687"
Oct 30 22:39:09.950824 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.950800    4947 scope.go:122] "RemoveContainer" containerID="2ca9042fa6c41a29928d5e14fdd2dbfd4b33996d393b1748a9df2bfd41372f2d"
Oct 30 22:39:09.959883 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.959855    4947 scope.go:122] "RemoveContainer" containerID="8a3092cf922ebdfa438067ed00b60a6d0d74b0975d6a8a90bd8a0ca215daa6b7"
Oct 30 22:39:09.968422 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.968395    4947 scope.go:122] "RemoveContainer" containerID="ddf9060804483b8573be3cddae0cc36fe24b9cfa8b5b124d4e90d20df8675687"
Oct 30 22:39:09.968741 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:39:09.968712    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf9060804483b8573be3cddae0cc36fe24b9cfa8b5b124d4e90d20df8675687\": container with ID starting with ddf9060804483b8573be3cddae0cc36fe24b9cfa8b5b124d4e90d20df8675687 not found: ID does not exist" containerID="ddf9060804483b8573be3cddae0cc36fe24b9cfa8b5b124d4e90d20df8675687"
Oct 30 22:39:09.968834 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.968752    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf9060804483b8573be3cddae0cc36fe24b9cfa8b5b124d4e90d20df8675687"} err="failed to get container status \"ddf9060804483b8573be3cddae0cc36fe24b9cfa8b5b124d4e90d20df8675687\": rpc error: code = NotFound desc = could not find container \"ddf9060804483b8573be3cddae0cc36fe24b9cfa8b5b124d4e90d20df8675687\": container with ID starting with ddf9060804483b8573be3cddae0cc36fe24b9cfa8b5b124d4e90d20df8675687 not found: ID does not exist"
Oct 30 22:39:09.968834 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.968774    4947 scope.go:122] "RemoveContainer" containerID="2ca9042fa6c41a29928d5e14fdd2dbfd4b33996d393b1748a9df2bfd41372f2d"
Oct 30 22:39:09.969065 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:39:09.969045    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca9042fa6c41a29928d5e14fdd2dbfd4b33996d393b1748a9df2bfd41372f2d\": container with ID starting with 2ca9042fa6c41a29928d5e14fdd2dbfd4b33996d393b1748a9df2bfd41372f2d not found: ID does not exist" containerID="2ca9042fa6c41a29928d5e14fdd2dbfd4b33996d393b1748a9df2bfd41372f2d"
Oct 30 22:39:09.969127 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.969073    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca9042fa6c41a29928d5e14fdd2dbfd4b33996d393b1748a9df2bfd41372f2d"} err="failed to get container status \"2ca9042fa6c41a29928d5e14fdd2dbfd4b33996d393b1748a9df2bfd41372f2d\": rpc error: code = NotFound desc = could not find container \"2ca9042fa6c41a29928d5e14fdd2dbfd4b33996d393b1748a9df2bfd41372f2d\": container with ID starting with 2ca9042fa6c41a29928d5e14fdd2dbfd4b33996d393b1748a9df2bfd41372f2d not found: ID does not exist"
Oct 30 22:39:09.969127 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.969089    4947 scope.go:122] "RemoveContainer" containerID="8a3092cf922ebdfa438067ed00b60a6d0d74b0975d6a8a90bd8a0ca215daa6b7"
Oct 30 22:39:09.969414 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:39:09.969388    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a3092cf922ebdfa438067ed00b60a6d0d74b0975d6a8a90bd8a0ca215daa6b7\": container with ID starting with 8a3092cf922ebdfa438067ed00b60a6d0d74b0975d6a8a90bd8a0ca215daa6b7 not found: ID does not exist" containerID="8a3092cf922ebdfa438067ed00b60a6d0d74b0975d6a8a90bd8a0ca215daa6b7"
Oct 30 22:39:09.969543 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:09.969418    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3092cf922ebdfa438067ed00b60a6d0d74b0975d6a8a90bd8a0ca215daa6b7"} err="failed to get container status \"8a3092cf922ebdfa438067ed00b60a6d0d74b0975d6a8a90bd8a0ca215daa6b7\": rpc error: code = NotFound desc = could not find container \"8a3092cf922ebdfa438067ed00b60a6d0d74b0975d6a8a90bd8a0ca215daa6b7\": container with ID starting with 8a3092cf922ebdfa438067ed00b60a6d0d74b0975d6a8a90bd8a0ca215daa6b7 not found: ID does not exist"
Oct 30 22:39:10.109081 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:10.109041    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"]
Oct 30 22:39:10.116664 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:10.116633    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-runtime-predictor-86c5568d6c-279d9"]
Oct 30 22:39:10.160252 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:10.160202    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" path="/var/lib/kubelet/pods/45f8467b-804f-4cb8-b3c1-2bc430d9351e/volumes"
Oct 30 22:39:12.701491 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:12.701462    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"]
Oct 30 22:39:12.701965 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:12.701830    4947 cpu_manager.go:414] "containerMap: removing container" podUID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" containerName="kube-rbac-proxy"
Oct 30 22:39:12.701965 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:12.701843    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" containerName="kube-rbac-proxy"
Oct 30 22:39:12.701965 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:12.701875    4947 cpu_manager.go:414] "containerMap: removing container" podUID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" containerName="storage-initializer"
Oct 30 22:39:12.701965 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:12.701883    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" containerName="storage-initializer"
Oct 30 22:39:12.701965 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:12.701893    4947 cpu_manager.go:414] "containerMap: removing container" podUID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" containerName="kserve-container"
Oct 30 22:39:12.701965 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:12.701900    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" containerName="kserve-container"
Oct 30 22:39:12.702236 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:12.701976    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" containerName="kserve-container"
Oct 30 22:39:12.702236 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:12.701989    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f8467b-804f-4cb8-b3c1-2bc430d9351e" containerName="kube-rbac-proxy"
Oct 30 22:39:12.964009 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:12.963936    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"]
Oct 30 22:39:12.964170 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:12.964063    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:13.020526 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.020496    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:13.020696 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.020547    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:13.020696 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.020642    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrpmm\" (UniqueName: \"kubernetes.io/projected/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kube-api-access-jrpmm\") pod \"isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:13.020803 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.020738    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:13.020803 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.020765    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:13.067276 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.067138    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:39:13.067276 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.067227    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Oct 30 22:39:13.067276 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.067235    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:39:13.067276 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.067242    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:39:13.067276 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.067254    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:39:13.067276 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.067264    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Oct 30 22:39:13.067276 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.067274    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:39:13.121513 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.121474    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrpmm\" (UniqueName: \"kubernetes.io/projected/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kube-api-access-jrpmm\") pod \"isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:13.121700 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.121536    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:13.121700 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.121567    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:13.121700 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.121609    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:13.121700 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.121667    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:13.122130 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.122080    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:13.122440 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.122415    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:13.122556 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.122459    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:13.124649 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.124626    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:13.185472 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.185438    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrpmm\" (UniqueName: \"kubernetes.io/projected/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kube-api-access-jrpmm\") pod \"isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:13.327395 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.327360    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:13.769386 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.769269    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"]
Oct 30 22:39:13.773043 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:39:13.773003    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaac4ef9_6e9c_4ed1_8ecb_abbfa66e7158.slice/crio-7b0aa96b591b2ac92e0c3596a41e9ca4afcae6ae22b720c1dba3ae42c5ce93d2 WatchSource:0}: Error finding container 7b0aa96b591b2ac92e0c3596a41e9ca4afcae6ae22b720c1dba3ae42c5ce93d2: Status 404 returned error can't find the container with id 7b0aa96b591b2ac92e0c3596a41e9ca4afcae6ae22b720c1dba3ae42c5ce93d2
Oct 30 22:39:13.961746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.961709    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" event={"ID":"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158","Type":"ContainerStarted","Data":"2eee44986eda7c2716fa15d7045197d61252bd7172145c2e54c52edf54cd5283"}
Oct 30 22:39:13.962010 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:13.961779    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" event={"ID":"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158","Type":"ContainerStarted","Data":"7b0aa96b591b2ac92e0c3596a41e9ca4afcae6ae22b720c1dba3ae42c5ce93d2"}
Oct 30 22:39:18.988430 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:18.988393    4947 generic.go:356] "Generic (PLEG): container finished" podID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerID="2eee44986eda7c2716fa15d7045197d61252bd7172145c2e54c52edf54cd5283" exitCode=0
Oct 30 22:39:18.988875 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:18.988476    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" event={"ID":"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158","Type":"ContainerDied","Data":"2eee44986eda7c2716fa15d7045197d61252bd7172145c2e54c52edf54cd5283"}
Oct 30 22:39:19.999946 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:19.999905    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" event={"ID":"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158","Type":"ContainerStarted","Data":"2523059a9a6f13f0dcdc229bc1ebb973afdbad3178dafd88921d19fcbf4732e7"}
Oct 30 22:39:20.000315 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:19.999979    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" event={"ID":"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158","Type":"ContainerStarted","Data":"dd4fa2d4e92799ef0598b418c720bc92240bf3f24faf56cb28517defd9641fb8"}
Oct 30 22:39:20.001022 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:20.000998    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:20.001093 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:20.001054    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:20.002361 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:20.002303    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 22:39:21.005775 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:21.005733    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 22:39:26.011450 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:26.011419    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:39:26.012219 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:26.012184    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 22:39:26.184361 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:26.184293    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" podStartSLOduration=10.045689236 podStartE2EDuration="14.184276569s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.138587333s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:20.123553777 +0000 UTC m=+1858.122901344" watchObservedRunningTime="2026-10-30 22:39:26.184276569 +0000 UTC m=+1864.183624160"
Oct 30 22:39:29.113686 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:39:29.113648    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:39:36.012625 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:36.012581    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 22:39:46.012479 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:46.012375    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 22:39:46.125371 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:46.125319    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4" event={"ID":"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b","Type":"ContainerStarted","Data":"d484e86821c795eec9e70fa680d9c7e7811ca27e005d66c2565a31b7369a51cf"}
Oct 30 22:39:46.125607 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:46.125402    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4" event={"ID":"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b","Type":"ContainerStarted","Data":"f1f631b78c5add47e046c1d533b8fb2ef0bb5ad3d71e8c8335bf48a805d7fcbf"}
Oct 30 22:39:46.125607 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:46.125593    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:39:46.125727 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:46.125634    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:39:46.127229 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:46.127197    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4" podUID="9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Oct 30 22:39:46.300523 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:46.299534    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4" podStartSLOduration=2.3298916849999998 podStartE2EDuration="1m56.299515867s" totalImagesPullingTime="1m48.08079174s" totalInitContainerRuntime="5.888832442s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:46.297067235 +0000 UTC m=+1884.296414822" watchObservedRunningTime="2026-10-30 22:39:46.299515867 +0000 UTC m=+1884.298863431"
Oct 30 22:39:47.129815 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:47.129771    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4" podUID="9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Oct 30 22:39:52.134955 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:52.134917    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:39:52.135563 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:52.135537    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4" podUID="9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Oct 30 22:39:56.012546 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:39:56.012506    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 22:39:59.047939 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:39:59.047899    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:40:02.135978 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:02.135948    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:40:06.012226 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:06.012187    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 22:40:16.012924 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:16.012887    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 22:40:17.634234 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:17.634200    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"]
Oct 30 22:40:17.634690 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:17.634473    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4" podUID="9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" containerName="kserve-container" containerID="cri-o://f1f631b78c5add47e046c1d533b8fb2ef0bb5ad3d71e8c8335bf48a805d7fcbf" gracePeriod=30
Oct 30 22:40:17.634690 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:17.634495    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4" podUID="9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" containerName="kube-rbac-proxy" containerID="cri-o://d484e86821c795eec9e70fa680d9c7e7811ca27e005d66c2565a31b7369a51cf" gracePeriod=30
Oct 30 22:40:18.263061 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:18.263026    4947 generic.go:356] "Generic (PLEG): container finished" podID="9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" containerID="d484e86821c795eec9e70fa680d9c7e7811ca27e005d66c2565a31b7369a51cf" exitCode=0
Oct 30 22:40:18.263245 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:18.263102    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4" event={"ID":"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b","Type":"ContainerDied","Data":"d484e86821c795eec9e70fa680d9c7e7811ca27e005d66c2565a31b7369a51cf"}
Oct 30 22:40:18.997495 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:18.997458    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:40:19.081030 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.080964    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kube-api-access-p85gk\" (UniqueName: \"kubernetes.io/projected/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kube-api-access-p85gk\") pod \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") "
Oct 30 22:40:19.081030 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.081021    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-cabundle-cert\") pod \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") "
Oct 30 22:40:19.081206 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.081051    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kserve-provision-location\") pod \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") "
Oct 30 22:40:19.081206 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.081080    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kube-rbac-proxy-sar-config\") pod \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") "
Oct 30 22:40:19.081206 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.081121    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-proxy-tls\") pod \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\" (UID: \"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b\") "
Oct 30 22:40:19.081518 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.081482    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-cabundle-cert" pod "9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" (UID: "9720ffe2-bc34-4367-b7b9-67b4a1a60a0b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:19.081518 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.081504    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kube-rbac-proxy-sar-config" pod "9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" (UID: "9720ffe2-bc34-4367-b7b9-67b4a1a60a0b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:19.083325 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.083293    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-proxy-tls" pod "9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" (UID: "9720ffe2-bc34-4367-b7b9-67b4a1a60a0b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:19.083428 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.083386    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kube-api-access-p85gk" pod "9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" (UID: "9720ffe2-bc34-4367-b7b9-67b4a1a60a0b"). InnerVolumeSpecName "kube-api-access-p85gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:19.084651 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.084630    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kserve-provision-location" pod "9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" (UID: "9720ffe2-bc34-4367-b7b9-67b4a1a60a0b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:19.182451 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.182415    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:19.182451 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.182440    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:19.182451 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.182450    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:19.182451 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.182459    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p85gk\" (UniqueName: \"kubernetes.io/projected/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-kube-api-access-p85gk\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:19.182701 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.182468    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:19.270142 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.270112    4947 generic.go:356] "Generic (PLEG): container finished" podID="9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" containerID="f1f631b78c5add47e046c1d533b8fb2ef0bb5ad3d71e8c8335bf48a805d7fcbf" exitCode=0
Oct 30 22:40:19.270319 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.270159    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4" event={"ID":"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b","Type":"ContainerDied","Data":"f1f631b78c5add47e046c1d533b8fb2ef0bb5ad3d71e8c8335bf48a805d7fcbf"}
Oct 30 22:40:19.270319 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.270179    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"
Oct 30 22:40:19.270319 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.270206    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4" event={"ID":"9720ffe2-bc34-4367-b7b9-67b4a1a60a0b","Type":"ContainerDied","Data":"4cb82a1a086bb950962e1ec6b481b09fe46d2707394d7eefd65b9748f7ab185d"}
Oct 30 22:40:19.270319 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.270232    4947 scope.go:122] "RemoveContainer" containerID="d484e86821c795eec9e70fa680d9c7e7811ca27e005d66c2565a31b7369a51cf"
Oct 30 22:40:19.279131 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.279114    4947 scope.go:122] "RemoveContainer" containerID="f1f631b78c5add47e046c1d533b8fb2ef0bb5ad3d71e8c8335bf48a805d7fcbf"
Oct 30 22:40:19.286967 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.286949    4947 scope.go:122] "RemoveContainer" containerID="25a8ebf51d5c6e5d8d17975d88679568daaff635bc9d03045ce5d4630f7ff1ae"
Oct 30 22:40:19.294913 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.294892    4947 scope.go:122] "RemoveContainer" containerID="d484e86821c795eec9e70fa680d9c7e7811ca27e005d66c2565a31b7369a51cf"
Oct 30 22:40:19.295172 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:40:19.295154    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d484e86821c795eec9e70fa680d9c7e7811ca27e005d66c2565a31b7369a51cf\": container with ID starting with d484e86821c795eec9e70fa680d9c7e7811ca27e005d66c2565a31b7369a51cf not found: ID does not exist" containerID="d484e86821c795eec9e70fa680d9c7e7811ca27e005d66c2565a31b7369a51cf"
Oct 30 22:40:19.295221 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.295182    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d484e86821c795eec9e70fa680d9c7e7811ca27e005d66c2565a31b7369a51cf"} err="failed to get container status \"d484e86821c795eec9e70fa680d9c7e7811ca27e005d66c2565a31b7369a51cf\": rpc error: code = NotFound desc = could not find container \"d484e86821c795eec9e70fa680d9c7e7811ca27e005d66c2565a31b7369a51cf\": container with ID starting with d484e86821c795eec9e70fa680d9c7e7811ca27e005d66c2565a31b7369a51cf not found: ID does not exist"
Oct 30 22:40:19.295221 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.295200    4947 scope.go:122] "RemoveContainer" containerID="f1f631b78c5add47e046c1d533b8fb2ef0bb5ad3d71e8c8335bf48a805d7fcbf"
Oct 30 22:40:19.295484 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:40:19.295469    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f631b78c5add47e046c1d533b8fb2ef0bb5ad3d71e8c8335bf48a805d7fcbf\": container with ID starting with f1f631b78c5add47e046c1d533b8fb2ef0bb5ad3d71e8c8335bf48a805d7fcbf not found: ID does not exist" containerID="f1f631b78c5add47e046c1d533b8fb2ef0bb5ad3d71e8c8335bf48a805d7fcbf"
Oct 30 22:40:19.295545 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.295488    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f631b78c5add47e046c1d533b8fb2ef0bb5ad3d71e8c8335bf48a805d7fcbf"} err="failed to get container status \"f1f631b78c5add47e046c1d533b8fb2ef0bb5ad3d71e8c8335bf48a805d7fcbf\": rpc error: code = NotFound desc = could not find container \"f1f631b78c5add47e046c1d533b8fb2ef0bb5ad3d71e8c8335bf48a805d7fcbf\": container with ID starting with f1f631b78c5add47e046c1d533b8fb2ef0bb5ad3d71e8c8335bf48a805d7fcbf not found: ID does not exist"
Oct 30 22:40:19.295545 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.295501    4947 scope.go:122] "RemoveContainer" containerID="25a8ebf51d5c6e5d8d17975d88679568daaff635bc9d03045ce5d4630f7ff1ae"
Oct 30 22:40:19.295764 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:40:19.295749    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a8ebf51d5c6e5d8d17975d88679568daaff635bc9d03045ce5d4630f7ff1ae\": container with ID starting with 25a8ebf51d5c6e5d8d17975d88679568daaff635bc9d03045ce5d4630f7ff1ae not found: ID does not exist" containerID="25a8ebf51d5c6e5d8d17975d88679568daaff635bc9d03045ce5d4630f7ff1ae"
Oct 30 22:40:19.295806 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.295769    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a8ebf51d5c6e5d8d17975d88679568daaff635bc9d03045ce5d4630f7ff1ae"} err="failed to get container status \"25a8ebf51d5c6e5d8d17975d88679568daaff635bc9d03045ce5d4630f7ff1ae\": rpc error: code = NotFound desc = could not find container \"25a8ebf51d5c6e5d8d17975d88679568daaff635bc9d03045ce5d4630f7ff1ae\": container with ID starting with 25a8ebf51d5c6e5d8d17975d88679568daaff635bc9d03045ce5d4630f7ff1ae not found: ID does not exist"
Oct 30 22:40:19.440049 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.439979    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"]
Oct 30 22:40:19.445945 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:19.445923    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-v1-predictor-5f5cb85fc9-rp2f4"]
Oct 30 22:40:20.157973 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:20.157939    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" path="/var/lib/kubelet/pods/9720ffe2-bc34-4367-b7b9-67b4a1a60a0b/volumes"
Oct 30 22:40:22.750821 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.750739    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"]
Oct 30 22:40:22.751187 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.751055    4947 cpu_manager.go:414] "containerMap: removing container" podUID="9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" containerName="kube-rbac-proxy"
Oct 30 22:40:22.751187 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.751065    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" containerName="kube-rbac-proxy"
Oct 30 22:40:22.751187 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.751089    4947 cpu_manager.go:414] "containerMap: removing container" podUID="9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" containerName="storage-initializer"
Oct 30 22:40:22.751187 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.751094    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" containerName="storage-initializer"
Oct 30 22:40:22.751187 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.751104    4947 cpu_manager.go:414] "containerMap: removing container" podUID="9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" containerName="kserve-container"
Oct 30 22:40:22.751187 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.751109    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" containerName="kserve-container"
Oct 30 22:40:22.751187 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.751168    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" containerName="kube-rbac-proxy"
Oct 30 22:40:22.751187 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.751179    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9720ffe2-bc34-4367-b7b9-67b4a1a60a0b" containerName="kserve-container"
Oct 30 22:40:22.755552 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.755529    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:22.863104 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.863076    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"openshift-service-ca.crt\""
Oct 30 22:40:22.863386 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.863362    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"seaweedfs-s3-creds\""
Oct 30 22:40:22.863517 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.863409    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:40:22.863517 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.863430    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"default-dockercfg-s9cqw\""
Oct 30 22:40:22.863517 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.863437    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"kube-root-ca.crt\""
Oct 30 22:40:22.863517 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.863446    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Oct 30 22:40:22.863700 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.863614    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Oct 30 22:40:22.865158 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.865136    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"]
Oct 30 22:40:22.906377 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.906348    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz2dn\" (UniqueName: \"kubernetes.io/projected/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kube-api-access-kz2dn\") pod \"isvc-autogluon-v2-predictor-5b668cb457-qv89x\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:22.906624 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.906385    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-5b668cb457-qv89x\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:22.906624 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.906405    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-5b668cb457-qv89x\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:22.906624 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.906422    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-5b668cb457-qv89x\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:22.906624 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:22.906537    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-proxy-tls\") pod \"isvc-autogluon-v2-predictor-5b668cb457-qv89x\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:23.007447 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:23.007378    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz2dn\" (UniqueName: \"kubernetes.io/projected/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kube-api-access-kz2dn\") pod \"isvc-autogluon-v2-predictor-5b668cb457-qv89x\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:23.007447 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:23.007416    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-5b668cb457-qv89x\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:23.007447 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:23.007442    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-5b668cb457-qv89x\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:23.007723 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:23.007552    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-5b668cb457-qv89x\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:23.007723 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:23.007601    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-proxy-tls\") pod \"isvc-autogluon-v2-predictor-5b668cb457-qv89x\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:23.007852 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:23.007835    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-5b668cb457-qv89x\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:23.008178 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:23.008156    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-5b668cb457-qv89x\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:23.008258 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:23.008242    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-5b668cb457-qv89x\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:23.010122 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:23.010106    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-proxy-tls\") pod \"isvc-autogluon-v2-predictor-5b668cb457-qv89x\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:23.070533 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:23.070506    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz2dn\" (UniqueName: \"kubernetes.io/projected/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kube-api-access-kz2dn\") pod \"isvc-autogluon-v2-predictor-5b668cb457-qv89x\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:23.071282 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:23.071259    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:23.498168 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:23.498104    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"]
Oct 30 22:40:23.500546 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:40:23.500516    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ad275a_8f0b_4c9f_9a79_34afeec3fba3.slice/crio-bd8cd09ee3dfb211d3f3b1d5cda83921a8ace4b8b87674508e83d23932c94c27 WatchSource:0}: Error finding container bd8cd09ee3dfb211d3f3b1d5cda83921a8ace4b8b87674508e83d23932c94c27: Status 404 returned error can't find the container with id bd8cd09ee3dfb211d3f3b1d5cda83921a8ace4b8b87674508e83d23932c94c27
Oct 30 22:40:24.293167 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:24.293131    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x" event={"ID":"48ad275a-8f0b-4c9f-9a79-34afeec3fba3","Type":"ContainerStarted","Data":"bcb043712fe24b5c7455edd08b4d222662ee5f75820c02bfff4d6d1aa6924fd3"}
Oct 30 22:40:24.293679 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:24.293186    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x" event={"ID":"48ad275a-8f0b-4c9f-9a79-34afeec3fba3","Type":"ContainerStarted","Data":"bd8cd09ee3dfb211d3f3b1d5cda83921a8ace4b8b87674508e83d23932c94c27"}
Oct 30 22:40:26.012576 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:26.012536    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 22:40:29.049494 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:40:29.049455    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:40:29.314620 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:29.314548    4947 generic.go:356] "Generic (PLEG): container finished" podID="48ad275a-8f0b-4c9f-9a79-34afeec3fba3" containerID="bcb043712fe24b5c7455edd08b4d222662ee5f75820c02bfff4d6d1aa6924fd3" exitCode=0
Oct 30 22:40:29.314620 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:29.314595    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x" event={"ID":"48ad275a-8f0b-4c9f-9a79-34afeec3fba3","Type":"ContainerDied","Data":"bcb043712fe24b5c7455edd08b4d222662ee5f75820c02bfff4d6d1aa6924fd3"}
Oct 30 22:40:30.320482 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:30.320446    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x" event={"ID":"48ad275a-8f0b-4c9f-9a79-34afeec3fba3","Type":"ContainerStarted","Data":"51e19bdd3aabac13024997367eeb7e2ebf772568fee0dd22128f46f79d0b6360"}
Oct 30 22:40:30.320870 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:30.320509    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x" event={"ID":"48ad275a-8f0b-4c9f-9a79-34afeec3fba3","Type":"ContainerStarted","Data":"de84e646b49eb05d096be3e20545aaf9fccc1a71477c83b08f7b724d70e89862"}
Oct 30 22:40:30.320870 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:30.320711    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:30.320870 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:30.320754    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:30.443213 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:30.443151    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x" podStartSLOduration=2.861181593 podStartE2EDuration="8.443133218s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.581951625s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:30.442504323 +0000 UTC m=+1928.441851923" watchObservedRunningTime="2026-10-30 22:40:30.443133218 +0000 UTC m=+1928.442480782"
Oct 30 22:40:36.012977 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:36.012899    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 22:40:36.330235 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:36.330206    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:40:44.158081 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:44.158049    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:40:53.764635 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:53.764595    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"]
Oct 30 22:40:53.765155 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:53.764834    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kserve-container" containerID="cri-o://dd4fa2d4e92799ef0598b418c720bc92240bf3f24faf56cb28517defd9641fb8" gracePeriod=30
Oct 30 22:40:53.765155 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:53.764913    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kube-rbac-proxy" containerID="cri-o://2523059a9a6f13f0dcdc229bc1ebb973afdbad3178dafd88921d19fcbf4732e7" gracePeriod=30
Oct 30 22:40:54.153420 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:54.153377    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 22:40:54.431784 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:54.431705    4947 generic.go:356] "Generic (PLEG): container finished" podID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerID="2523059a9a6f13f0dcdc229bc1ebb973afdbad3178dafd88921d19fcbf4732e7" exitCode=0
Oct 30 22:40:54.431930 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:54.431781    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" event={"ID":"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158","Type":"ContainerDied","Data":"2523059a9a6f13f0dcdc229bc1ebb973afdbad3178dafd88921d19fcbf4732e7"}
Oct 30 22:40:56.006752 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:56.006703    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Oct 30 22:40:58.454207 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.454168    4947 generic.go:356] "Generic (PLEG): container finished" podID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerID="dd4fa2d4e92799ef0598b418c720bc92240bf3f24faf56cb28517defd9641fb8" exitCode=0
Oct 30 22:40:58.454618 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.454204    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" event={"ID":"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158","Type":"ContainerDied","Data":"dd4fa2d4e92799ef0598b418c720bc92240bf3f24faf56cb28517defd9641fb8"}
Oct 30 22:40:58.503674 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.503651    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:40:58.596687 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.596607    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-cabundle-cert\") pod \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") "
Oct 30 22:40:58.596687 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.596670    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kube-api-access-jrpmm\" (UniqueName: \"kubernetes.io/projected/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kube-api-access-jrpmm\") pod \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") "
Oct 30 22:40:58.596886 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.596716    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kserve-provision-location\") pod \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") "
Oct 30 22:40:58.596886 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.596748    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-proxy-tls\") pod \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") "
Oct 30 22:40:58.596886 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.596773    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kube-rbac-proxy-sar-config\") pod \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\" (UID: \"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158\") "
Oct 30 22:40:58.597024 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.596996    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-cabundle-cert" pod "eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" (UID: "eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:58.597066 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.597018    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kserve-provision-location" pod "eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" (UID: "eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:58.597193 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.597170    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kube-rbac-proxy-sar-config" pod "eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" (UID: "eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:58.598926 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.598900    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-proxy-tls" pod "eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" (UID: "eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:58.599019 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.599003    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kube-api-access-jrpmm" pod "eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" (UID: "eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158"). InnerVolumeSpecName "kube-api-access-jrpmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:58.697513 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.697482    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:58.697513 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.697508    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrpmm\" (UniqueName: \"kubernetes.io/projected/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kube-api-access-jrpmm\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:58.697513 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.697519    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:58.697725 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.697529    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:58.697725 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:58.697538    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:59.047759 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:40:59.047719    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:40:59.460687 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:59.460611    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"
Oct 30 22:40:59.460687 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:59.460633    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg" event={"ID":"eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158","Type":"ContainerDied","Data":"7b0aa96b591b2ac92e0c3596a41e9ca4afcae6ae22b720c1dba3ae42c5ce93d2"}
Oct 30 22:40:59.461148 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:59.460712    4947 scope.go:122] "RemoveContainer" containerID="2523059a9a6f13f0dcdc229bc1ebb973afdbad3178dafd88921d19fcbf4732e7"
Oct 30 22:40:59.473134 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:59.473110    4947 scope.go:122] "RemoveContainer" containerID="dd4fa2d4e92799ef0598b418c720bc92240bf3f24faf56cb28517defd9641fb8"
Oct 30 22:40:59.481116 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:59.481093    4947 scope.go:122] "RemoveContainer" containerID="2eee44986eda7c2716fa15d7045197d61252bd7172145c2e54c52edf54cd5283"
Oct 30 22:40:59.628598 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:59.628565    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"]
Oct 30 22:40:59.678261 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:40:59.678231    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-predictor-7d7fdb4bdc-4zflg"]
Oct 30 22:41:00.158205 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:00.158168    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" path="/var/lib/kubelet/pods/eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158/volumes"
Oct 30 22:41:02.851249 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.851214    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"]
Oct 30 22:41:02.851645 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.851583    4947 cpu_manager.go:414] "containerMap: removing container" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="storage-initializer"
Oct 30 22:41:02.851645 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.851598    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="storage-initializer"
Oct 30 22:41:02.851645 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.851609    4947 cpu_manager.go:414] "containerMap: removing container" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kube-rbac-proxy"
Oct 30 22:41:02.851645 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.851614    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kube-rbac-proxy"
Oct 30 22:41:02.851645 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.851627    4947 cpu_manager.go:414] "containerMap: removing container" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kserve-container"
Oct 30 22:41:02.851645 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.851632    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kserve-container"
Oct 30 22:41:02.851831 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.851705    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kube-rbac-proxy"
Oct 30 22:41:02.851831 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.851713    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaac4ef9-6e9c-4ed1-8ecb-abbfa66e7158" containerName="kserve-container"
Oct 30 22:41:02.854546 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.854528    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:02.910355 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.910318    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:41:02.910355 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.910322    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:41:02.929220 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.929190    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0acf659-04ea-42ad-9232-d99b2e243be3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:02.929327 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.929229    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0acf659-04ea-42ad-9232-d99b2e243be3-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:02.929327 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.929256    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xwqc\" (UniqueName: \"kubernetes.io/projected/a0acf659-04ea-42ad-9232-d99b2e243be3-kube-api-access-4xwqc\") pod \"isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:02.929327 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.929305    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0acf659-04ea-42ad-9232-d99b2e243be3-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:02.929471 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.929414    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0acf659-04ea-42ad-9232-d99b2e243be3-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:02.960439 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.960407    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:41:02.960439 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.960433    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Oct 30 22:41:02.960439 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.960407    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:41:02.960700 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.960473    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:41:02.960700 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.960500    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Oct 30 22:41:02.961859 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:02.961838    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"]
Oct 30 22:41:03.030569 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:03.030544    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0acf659-04ea-42ad-9232-d99b2e243be3-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:03.030681 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:03.030583    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xwqc\" (UniqueName: \"kubernetes.io/projected/a0acf659-04ea-42ad-9232-d99b2e243be3-kube-api-access-4xwqc\") pod \"isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:03.030681 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:03.030600    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0acf659-04ea-42ad-9232-d99b2e243be3-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:03.030681 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:03.030624    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0acf659-04ea-42ad-9232-d99b2e243be3-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:03.030681 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:03.030663    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0acf659-04ea-42ad-9232-d99b2e243be3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:03.031123 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:03.031101    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0acf659-04ea-42ad-9232-d99b2e243be3-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:03.031325 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:03.031308    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0acf659-04ea-42ad-9232-d99b2e243be3-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:03.031412 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:03.031316    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0acf659-04ea-42ad-9232-d99b2e243be3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:03.033227 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:03.033208    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0acf659-04ea-42ad-9232-d99b2e243be3-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:03.094025 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:03.093995    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xwqc\" (UniqueName: \"kubernetes.io/projected/a0acf659-04ea-42ad-9232-d99b2e243be3-kube-api-access-4xwqc\") pod \"isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:03.167086 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:03.167020    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:03.589500 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:03.589479    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"]
Oct 30 22:41:03.592296 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:41:03.592269    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0acf659_04ea_42ad_9232_d99b2e243be3.slice/crio-aa05321634c921d29ccf615d28cb65210c8b8ada1e70b748c1f19dfc4d7090fa WatchSource:0}: Error finding container aa05321634c921d29ccf615d28cb65210c8b8ada1e70b748c1f19dfc4d7090fa: Status 404 returned error can't find the container with id aa05321634c921d29ccf615d28cb65210c8b8ada1e70b748c1f19dfc4d7090fa
Oct 30 22:41:04.483155 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:04.483118    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" event={"ID":"a0acf659-04ea-42ad-9232-d99b2e243be3","Type":"ContainerStarted","Data":"1523d3f9780a03f844f669d280f277b5323f3678299363e4992d98637be58173"}
Oct 30 22:41:04.483580 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:04.483180    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" event={"ID":"a0acf659-04ea-42ad-9232-d99b2e243be3","Type":"ContainerStarted","Data":"aa05321634c921d29ccf615d28cb65210c8b8ada1e70b748c1f19dfc4d7090fa"}
Oct 30 22:41:08.501219 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:08.501181    4947 generic.go:356] "Generic (PLEG): container finished" podID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerID="1523d3f9780a03f844f669d280f277b5323f3678299363e4992d98637be58173" exitCode=0
Oct 30 22:41:08.501655 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:08.501255    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" event={"ID":"a0acf659-04ea-42ad-9232-d99b2e243be3","Type":"ContainerDied","Data":"1523d3f9780a03f844f669d280f277b5323f3678299363e4992d98637be58173"}
Oct 30 22:41:09.508005 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:09.507974    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" event={"ID":"a0acf659-04ea-42ad-9232-d99b2e243be3","Type":"ContainerStarted","Data":"a1536e88f8cc84346f8874fef5ee3b4fbc6b9c3a598e22834665c38264c15b83"}
Oct 30 22:41:09.508418 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:09.508033    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" event={"ID":"a0acf659-04ea-42ad-9232-d99b2e243be3","Type":"ContainerStarted","Data":"0da6bfb7911878d0f98ab00b7e95adb34cdea922808cbcc32baadf340c729c6b"}
Oct 30 22:41:09.508418 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:09.508216    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:09.630191 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:09.630142    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" podStartSLOduration=3.6958567159999998 podStartE2EDuration="7.630129266s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.93427255s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:09.628676826 +0000 UTC m=+1967.628024391" watchObservedRunningTime="2026-10-30 22:41:09.630129266 +0000 UTC m=+1967.629476828"
Oct 30 22:41:10.512612 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:10.512583    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:10.513977 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:10.513951    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Oct 30 22:41:11.522249 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:11.522203    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Oct 30 22:41:16.527208 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:16.527177    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:41:16.527658 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:16.527635    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Oct 30 22:41:26.528496 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:26.528454    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Oct 30 22:41:29.047794 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:41:29.047747    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:41:36.528167 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:36.528121    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Oct 30 22:41:46.528292 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:46.528250    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Oct 30 22:41:56.528278 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:56.528241    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Oct 30 22:41:59.047954 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:41:59.047920    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:41:59.185325 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:41:59.185289    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:42:06.528546 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:06.528459    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Oct 30 22:42:09.009819 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:09.009781    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"]
Oct 30 22:42:09.010257 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:09.010043    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x" podUID="48ad275a-8f0b-4c9f-9a79-34afeec3fba3" containerName="kserve-container" containerID="cri-o://de84e646b49eb05d096be3e20545aaf9fccc1a71477c83b08f7b724d70e89862" gracePeriod=30
Oct 30 22:42:09.010257 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:09.010057    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x" podUID="48ad275a-8f0b-4c9f-9a79-34afeec3fba3" containerName="kube-rbac-proxy" containerID="cri-o://51e19bdd3aabac13024997367eeb7e2ebf772568fee0dd22128f46f79d0b6360" gracePeriod=30
Oct 30 22:42:09.153261 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:09.153223    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x" podUID="48ad275a-8f0b-4c9f-9a79-34afeec3fba3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.49:8080/v2/models/isvc-autogluon-v2/ready\": dial tcp 10.132.0.49:8080: connect: connection refused"
Oct 30 22:42:09.763396 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:09.763307    4947 generic.go:356] "Generic (PLEG): container finished" podID="48ad275a-8f0b-4c9f-9a79-34afeec3fba3" containerID="51e19bdd3aabac13024997367eeb7e2ebf772568fee0dd22128f46f79d0b6360" exitCode=0
Oct 30 22:42:09.763396 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:09.763372    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x" event={"ID":"48ad275a-8f0b-4c9f-9a79-34afeec3fba3","Type":"ContainerDied","Data":"51e19bdd3aabac13024997367eeb7e2ebf772568fee0dd22128f46f79d0b6360"}
Oct 30 22:42:10.356142 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.356117    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:42:10.370392 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.370370    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kube-api-access-kz2dn\" (UniqueName: \"kubernetes.io/projected/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kube-api-access-kz2dn\") pod \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") "
Oct 30 22:42:10.370464 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.370409    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-cabundle-cert\") pod \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") "
Oct 30 22:42:10.370464 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.370452    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kube-rbac-proxy-sar-config\") pod \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") "
Oct 30 22:42:10.370546 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.370473    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-proxy-tls\" (UniqueName: \"kubernetes.io/secret/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-proxy-tls\") pod \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") "
Oct 30 22:42:10.370546 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.370495    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kserve-provision-location\") pod \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\" (UID: \"48ad275a-8f0b-4c9f-9a79-34afeec3fba3\") "
Oct 30 22:42:10.370855 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.370819    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kube-rbac-proxy-sar-config" pod "48ad275a-8f0b-4c9f-9a79-34afeec3fba3" (UID: "48ad275a-8f0b-4c9f-9a79-34afeec3fba3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:10.370921 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.370852    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-cabundle-cert" pod "48ad275a-8f0b-4c9f-9a79-34afeec3fba3" (UID: "48ad275a-8f0b-4c9f-9a79-34afeec3fba3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:10.372772 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.372749    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kube-api-access-kz2dn" pod "48ad275a-8f0b-4c9f-9a79-34afeec3fba3" (UID: "48ad275a-8f0b-4c9f-9a79-34afeec3fba3"). InnerVolumeSpecName "kube-api-access-kz2dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:10.373048 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.373027    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-proxy-tls" pod "48ad275a-8f0b-4c9f-9a79-34afeec3fba3" (UID: "48ad275a-8f0b-4c9f-9a79-34afeec3fba3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:10.374803 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.374770    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kserve-provision-location" pod "48ad275a-8f0b-4c9f-9a79-34afeec3fba3" (UID: "48ad275a-8f0b-4c9f-9a79-34afeec3fba3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:10.471632 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.471572    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:10.471632 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.471593    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:10.471632 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.471605    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:10.471632 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.471615    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz2dn\" (UniqueName: \"kubernetes.io/projected/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-kube-api-access-kz2dn\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:10.471632 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.471624    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48ad275a-8f0b-4c9f-9a79-34afeec3fba3-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:10.769859 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.769777    4947 generic.go:356] "Generic (PLEG): container finished" podID="48ad275a-8f0b-4c9f-9a79-34afeec3fba3" containerID="de84e646b49eb05d096be3e20545aaf9fccc1a71477c83b08f7b724d70e89862" exitCode=0
Oct 30 22:42:10.769859 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.769845    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"
Oct 30 22:42:10.770025 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.769858    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x" event={"ID":"48ad275a-8f0b-4c9f-9a79-34afeec3fba3","Type":"ContainerDied","Data":"de84e646b49eb05d096be3e20545aaf9fccc1a71477c83b08f7b724d70e89862"}
Oct 30 22:42:10.770025 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.769924    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x" event={"ID":"48ad275a-8f0b-4c9f-9a79-34afeec3fba3","Type":"ContainerDied","Data":"bd8cd09ee3dfb211d3f3b1d5cda83921a8ace4b8b87674508e83d23932c94c27"}
Oct 30 22:42:10.770025 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.769955    4947 scope.go:122] "RemoveContainer" containerID="51e19bdd3aabac13024997367eeb7e2ebf772568fee0dd22128f46f79d0b6360"
Oct 30 22:42:10.781159 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.781137    4947 scope.go:122] "RemoveContainer" containerID="de84e646b49eb05d096be3e20545aaf9fccc1a71477c83b08f7b724d70e89862"
Oct 30 22:42:10.788752 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.788735    4947 scope.go:122] "RemoveContainer" containerID="bcb043712fe24b5c7455edd08b4d222662ee5f75820c02bfff4d6d1aa6924fd3"
Oct 30 22:42:10.795777 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.795748    4947 scope.go:122] "RemoveContainer" containerID="51e19bdd3aabac13024997367eeb7e2ebf772568fee0dd22128f46f79d0b6360"
Oct 30 22:42:10.796030 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:42:10.796000    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e19bdd3aabac13024997367eeb7e2ebf772568fee0dd22128f46f79d0b6360\": container with ID starting with 51e19bdd3aabac13024997367eeb7e2ebf772568fee0dd22128f46f79d0b6360 not found: ID does not exist" containerID="51e19bdd3aabac13024997367eeb7e2ebf772568fee0dd22128f46f79d0b6360"
Oct 30 22:42:10.796106 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.796029    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e19bdd3aabac13024997367eeb7e2ebf772568fee0dd22128f46f79d0b6360"} err="failed to get container status \"51e19bdd3aabac13024997367eeb7e2ebf772568fee0dd22128f46f79d0b6360\": rpc error: code = NotFound desc = could not find container \"51e19bdd3aabac13024997367eeb7e2ebf772568fee0dd22128f46f79d0b6360\": container with ID starting with 51e19bdd3aabac13024997367eeb7e2ebf772568fee0dd22128f46f79d0b6360 not found: ID does not exist"
Oct 30 22:42:10.796106 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.796049    4947 scope.go:122] "RemoveContainer" containerID="de84e646b49eb05d096be3e20545aaf9fccc1a71477c83b08f7b724d70e89862"
Oct 30 22:42:10.796272 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:42:10.796257    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de84e646b49eb05d096be3e20545aaf9fccc1a71477c83b08f7b724d70e89862\": container with ID starting with de84e646b49eb05d096be3e20545aaf9fccc1a71477c83b08f7b724d70e89862 not found: ID does not exist" containerID="de84e646b49eb05d096be3e20545aaf9fccc1a71477c83b08f7b724d70e89862"
Oct 30 22:42:10.796360 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.796288    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de84e646b49eb05d096be3e20545aaf9fccc1a71477c83b08f7b724d70e89862"} err="failed to get container status \"de84e646b49eb05d096be3e20545aaf9fccc1a71477c83b08f7b724d70e89862\": rpc error: code = NotFound desc = could not find container \"de84e646b49eb05d096be3e20545aaf9fccc1a71477c83b08f7b724d70e89862\": container with ID starting with de84e646b49eb05d096be3e20545aaf9fccc1a71477c83b08f7b724d70e89862 not found: ID does not exist"
Oct 30 22:42:10.796360 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.796302    4947 scope.go:122] "RemoveContainer" containerID="bcb043712fe24b5c7455edd08b4d222662ee5f75820c02bfff4d6d1aa6924fd3"
Oct 30 22:42:10.796561 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:42:10.796540    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb043712fe24b5c7455edd08b4d222662ee5f75820c02bfff4d6d1aa6924fd3\": container with ID starting with bcb043712fe24b5c7455edd08b4d222662ee5f75820c02bfff4d6d1aa6924fd3 not found: ID does not exist" containerID="bcb043712fe24b5c7455edd08b4d222662ee5f75820c02bfff4d6d1aa6924fd3"
Oct 30 22:42:10.796622 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.796565    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb043712fe24b5c7455edd08b4d222662ee5f75820c02bfff4d6d1aa6924fd3"} err="failed to get container status \"bcb043712fe24b5c7455edd08b4d222662ee5f75820c02bfff4d6d1aa6924fd3\": rpc error: code = NotFound desc = could not find container \"bcb043712fe24b5c7455edd08b4d222662ee5f75820c02bfff4d6d1aa6924fd3\": container with ID starting with bcb043712fe24b5c7455edd08b4d222662ee5f75820c02bfff4d6d1aa6924fd3 not found: ID does not exist"
Oct 30 22:42:10.937885 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.937847    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"]
Oct 30 22:42:10.987646 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:10.987612    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-v2-predictor-5b668cb457-qv89x"]
Oct 30 22:42:12.162306 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:12.162268    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ad275a-8f0b-4c9f-9a79-34afeec3fba3" path="/var/lib/kubelet/pods/48ad275a-8f0b-4c9f-9a79-34afeec3fba3/volumes"
Oct 30 22:42:16.528354 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:16.528302    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Oct 30 22:42:24.157650 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:24.157620    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:42:29.048020 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:42:29.047987    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:42:33.699113 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:33.699079    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"]
Oct 30 22:42:33.699647 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:33.699325    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kserve-container" containerID="cri-o://0da6bfb7911878d0f98ab00b7e95adb34cdea922808cbcc32baadf340c729c6b" gracePeriod=30
Oct 30 22:42:33.699647 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:33.699402    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kube-rbac-proxy" containerID="cri-o://a1536e88f8cc84346f8874fef5ee3b4fbc6b9c3a598e22834665c38264c15b83" gracePeriod=30
Oct 30 22:42:33.865785 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:33.865757    4947 generic.go:356] "Generic (PLEG): container finished" podID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerID="a1536e88f8cc84346f8874fef5ee3b4fbc6b9c3a598e22834665c38264c15b83" exitCode=0
Oct 30 22:42:33.865945 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:33.865835    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" event={"ID":"a0acf659-04ea-42ad-9232-d99b2e243be3","Type":"ContainerDied","Data":"a1536e88f8cc84346f8874fef5ee3b4fbc6b9c3a598e22834665c38264c15b83"}
Oct 30 22:42:34.153469 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:34.153424    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Oct 30 22:42:36.523061 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:36.523007    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Oct 30 22:42:38.652104 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.652079    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:42:38.792531 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.792501    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0acf659-04ea-42ad-9232-d99b2e243be3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0acf659-04ea-42ad-9232-d99b2e243be3-kserve-provision-location\") pod \"a0acf659-04ea-42ad-9232-d99b2e243be3\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") "
Oct 30 22:42:38.792731 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.792571    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0acf659-04ea-42ad-9232-d99b2e243be3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0acf659-04ea-42ad-9232-d99b2e243be3-kube-rbac-proxy-sar-config\") pod \"a0acf659-04ea-42ad-9232-d99b2e243be3\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") "
Oct 30 22:42:38.792731 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.792594    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0acf659-04ea-42ad-9232-d99b2e243be3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0acf659-04ea-42ad-9232-d99b2e243be3-cabundle-cert\") pod \"a0acf659-04ea-42ad-9232-d99b2e243be3\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") "
Oct 30 22:42:38.792731 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.792639    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0acf659-04ea-42ad-9232-d99b2e243be3-kube-api-access-4xwqc\" (UniqueName: \"kubernetes.io/projected/a0acf659-04ea-42ad-9232-d99b2e243be3-kube-api-access-4xwqc\") pod \"a0acf659-04ea-42ad-9232-d99b2e243be3\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") "
Oct 30 22:42:38.792731 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.792659    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0acf659-04ea-42ad-9232-d99b2e243be3-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0acf659-04ea-42ad-9232-d99b2e243be3-proxy-tls\") pod \"a0acf659-04ea-42ad-9232-d99b2e243be3\" (UID: \"a0acf659-04ea-42ad-9232-d99b2e243be3\") "
Oct 30 22:42:38.793003 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.792897    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0acf659-04ea-42ad-9232-d99b2e243be3-kserve-provision-location" pod "a0acf659-04ea-42ad-9232-d99b2e243be3" (UID: "a0acf659-04ea-42ad-9232-d99b2e243be3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:38.793046 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.793013    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0acf659-04ea-42ad-9232-d99b2e243be3-kube-rbac-proxy-sar-config" pod "a0acf659-04ea-42ad-9232-d99b2e243be3" (UID: "a0acf659-04ea-42ad-9232-d99b2e243be3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:38.793046 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.793029    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0acf659-04ea-42ad-9232-d99b2e243be3-cabundle-cert" pod "a0acf659-04ea-42ad-9232-d99b2e243be3" (UID: "a0acf659-04ea-42ad-9232-d99b2e243be3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:38.794984 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.794960    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0acf659-04ea-42ad-9232-d99b2e243be3-kube-api-access-4xwqc" pod "a0acf659-04ea-42ad-9232-d99b2e243be3" (UID: "a0acf659-04ea-42ad-9232-d99b2e243be3"). InnerVolumeSpecName "kube-api-access-4xwqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:38.795054 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.795019    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0acf659-04ea-42ad-9232-d99b2e243be3-proxy-tls" pod "a0acf659-04ea-42ad-9232-d99b2e243be3" (UID: "a0acf659-04ea-42ad-9232-d99b2e243be3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:38.889247 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.889213    4947 generic.go:356] "Generic (PLEG): container finished" podID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerID="0da6bfb7911878d0f98ab00b7e95adb34cdea922808cbcc32baadf340c729c6b" exitCode=0
Oct 30 22:42:38.889452 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.889275    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"
Oct 30 22:42:38.889452 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.889296    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" event={"ID":"a0acf659-04ea-42ad-9232-d99b2e243be3","Type":"ContainerDied","Data":"0da6bfb7911878d0f98ab00b7e95adb34cdea922808cbcc32baadf340c729c6b"}
Oct 30 22:42:38.889452 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.889392    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6" event={"ID":"a0acf659-04ea-42ad-9232-d99b2e243be3","Type":"ContainerDied","Data":"aa05321634c921d29ccf615d28cb65210c8b8ada1e70b748c1f19dfc4d7090fa"}
Oct 30 22:42:38.889452 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.889430    4947 scope.go:122] "RemoveContainer" containerID="a1536e88f8cc84346f8874fef5ee3b4fbc6b9c3a598e22834665c38264c15b83"
Oct 30 22:42:38.893584 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.893561    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0acf659-04ea-42ad-9232-d99b2e243be3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:38.893688 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.893587    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0acf659-04ea-42ad-9232-d99b2e243be3-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:38.893688 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.893601    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xwqc\" (UniqueName: \"kubernetes.io/projected/a0acf659-04ea-42ad-9232-d99b2e243be3-kube-api-access-4xwqc\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:38.893688 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.893615    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0acf659-04ea-42ad-9232-d99b2e243be3-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:38.893688 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.893628    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0acf659-04ea-42ad-9232-d99b2e243be3-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:42:38.898396 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.898378    4947 scope.go:122] "RemoveContainer" containerID="0da6bfb7911878d0f98ab00b7e95adb34cdea922808cbcc32baadf340c729c6b"
Oct 30 22:42:38.905796 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.905777    4947 scope.go:122] "RemoveContainer" containerID="1523d3f9780a03f844f669d280f277b5323f3678299363e4992d98637be58173"
Oct 30 22:42:38.913007 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.912989    4947 scope.go:122] "RemoveContainer" containerID="a1536e88f8cc84346f8874fef5ee3b4fbc6b9c3a598e22834665c38264c15b83"
Oct 30 22:42:38.913305 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:42:38.913281    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1536e88f8cc84346f8874fef5ee3b4fbc6b9c3a598e22834665c38264c15b83\": container with ID starting with a1536e88f8cc84346f8874fef5ee3b4fbc6b9c3a598e22834665c38264c15b83 not found: ID does not exist" containerID="a1536e88f8cc84346f8874fef5ee3b4fbc6b9c3a598e22834665c38264c15b83"
Oct 30 22:42:38.913444 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.913311    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1536e88f8cc84346f8874fef5ee3b4fbc6b9c3a598e22834665c38264c15b83"} err="failed to get container status \"a1536e88f8cc84346f8874fef5ee3b4fbc6b9c3a598e22834665c38264c15b83\": rpc error: code = NotFound desc = could not find container \"a1536e88f8cc84346f8874fef5ee3b4fbc6b9c3a598e22834665c38264c15b83\": container with ID starting with a1536e88f8cc84346f8874fef5ee3b4fbc6b9c3a598e22834665c38264c15b83 not found: ID does not exist"
Oct 30 22:42:38.913444 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.913345    4947 scope.go:122] "RemoveContainer" containerID="0da6bfb7911878d0f98ab00b7e95adb34cdea922808cbcc32baadf340c729c6b"
Oct 30 22:42:38.913611 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:42:38.913580    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da6bfb7911878d0f98ab00b7e95adb34cdea922808cbcc32baadf340c729c6b\": container with ID starting with 0da6bfb7911878d0f98ab00b7e95adb34cdea922808cbcc32baadf340c729c6b not found: ID does not exist" containerID="0da6bfb7911878d0f98ab00b7e95adb34cdea922808cbcc32baadf340c729c6b"
Oct 30 22:42:38.913651 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.913619    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da6bfb7911878d0f98ab00b7e95adb34cdea922808cbcc32baadf340c729c6b"} err="failed to get container status \"0da6bfb7911878d0f98ab00b7e95adb34cdea922808cbcc32baadf340c729c6b\": rpc error: code = NotFound desc = could not find container \"0da6bfb7911878d0f98ab00b7e95adb34cdea922808cbcc32baadf340c729c6b\": container with ID starting with 0da6bfb7911878d0f98ab00b7e95adb34cdea922808cbcc32baadf340c729c6b not found: ID does not exist"
Oct 30 22:42:38.913651 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.913639    4947 scope.go:122] "RemoveContainer" containerID="1523d3f9780a03f844f669d280f277b5323f3678299363e4992d98637be58173"
Oct 30 22:42:38.913893 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:42:38.913879    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1523d3f9780a03f844f669d280f277b5323f3678299363e4992d98637be58173\": container with ID starting with 1523d3f9780a03f844f669d280f277b5323f3678299363e4992d98637be58173 not found: ID does not exist" containerID="1523d3f9780a03f844f669d280f277b5323f3678299363e4992d98637be58173"
Oct 30 22:42:38.913935 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:38.913898    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1523d3f9780a03f844f669d280f277b5323f3678299363e4992d98637be58173"} err="failed to get container status \"1523d3f9780a03f844f669d280f277b5323f3678299363e4992d98637be58173\": rpc error: code = NotFound desc = could not find container \"1523d3f9780a03f844f669d280f277b5323f3678299363e4992d98637be58173\": container with ID starting with 1523d3f9780a03f844f669d280f277b5323f3678299363e4992d98637be58173 not found: ID does not exist"
Oct 30 22:42:39.056993 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:39.056905    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"]
Oct 30 22:42:39.062646 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:39.062620    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-v2-mixed-predictor-5bf7f47b98-m5ql6"]
Oct 30 22:42:40.157746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:40.157707    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" path="/var/lib/kubelet/pods/a0acf659-04ea-42ad-9232-d99b2e243be3/volumes"
Oct 30 22:42:42.767433 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.767399    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"]
Oct 30 22:42:42.767806 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.767702    4947 cpu_manager.go:414] "containerMap: removing container" podUID="48ad275a-8f0b-4c9f-9a79-34afeec3fba3" containerName="kube-rbac-proxy"
Oct 30 22:42:42.767806 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.767712    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ad275a-8f0b-4c9f-9a79-34afeec3fba3" containerName="kube-rbac-proxy"
Oct 30 22:42:42.767806 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.767724    4947 cpu_manager.go:414] "containerMap: removing container" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kserve-container"
Oct 30 22:42:42.767806 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.767729    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kserve-container"
Oct 30 22:42:42.767806 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.767749    4947 cpu_manager.go:414] "containerMap: removing container" podUID="48ad275a-8f0b-4c9f-9a79-34afeec3fba3" containerName="kserve-container"
Oct 30 22:42:42.767806 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.767754    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ad275a-8f0b-4c9f-9a79-34afeec3fba3" containerName="kserve-container"
Oct 30 22:42:42.767806 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.767761    4947 cpu_manager.go:414] "containerMap: removing container" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="storage-initializer"
Oct 30 22:42:42.767806 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.767766    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="storage-initializer"
Oct 30 22:42:42.767806 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.767774    4947 cpu_manager.go:414] "containerMap: removing container" podUID="48ad275a-8f0b-4c9f-9a79-34afeec3fba3" containerName="storage-initializer"
Oct 30 22:42:42.767806 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.767779    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ad275a-8f0b-4c9f-9a79-34afeec3fba3" containerName="storage-initializer"
Oct 30 22:42:42.767806 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.767788    4947 cpu_manager.go:414] "containerMap: removing container" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kube-rbac-proxy"
Oct 30 22:42:42.767806 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.767793    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kube-rbac-proxy"
Oct 30 22:42:42.768231 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.767845    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ad275a-8f0b-4c9f-9a79-34afeec3fba3" containerName="kube-rbac-proxy"
Oct 30 22:42:42.768231 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.767854    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kserve-container"
Oct 30 22:42:42.768231 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.767862    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ad275a-8f0b-4c9f-9a79-34afeec3fba3" containerName="kserve-container"
Oct 30 22:42:42.768231 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.767869    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0acf659-04ea-42ad-9232-d99b2e243be3" containerName="kube-rbac-proxy"
Oct 30 22:42:42.772658 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.772641    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:42.822350 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.822296    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4zxl\" (UniqueName: \"kubernetes.io/projected/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kube-api-access-t4zxl\") pod \"isvc-tensorflow-predictor-d965dfcc-h7ph5\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:42.822500 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.822363    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-d965dfcc-h7ph5\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:42.822500 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.822473    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kserve-provision-location\") pod \"isvc-tensorflow-predictor-d965dfcc-h7ph5\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:42.822590 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.822539    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-proxy-tls\") pod \"isvc-tensorflow-predictor-d965dfcc-h7ph5\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:42.822641 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.822594    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-cabundle-cert\") pod \"isvc-tensorflow-predictor-d965dfcc-h7ph5\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:42.826720 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.826702    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:42:42.876273 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.876240    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Oct 30 22:42:42.876273 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.876245    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:42:42.876506 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.876296    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:42:42.876506 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.876328    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:42:42.876506 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.876327    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:42:42.876506 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.876412    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-tensorflow-predictor-serving-cert\""
Oct 30 22:42:42.878028 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.878008    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"]
Oct 30 22:42:42.923551 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.923514    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4zxl\" (UniqueName: \"kubernetes.io/projected/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kube-api-access-t4zxl\") pod \"isvc-tensorflow-predictor-d965dfcc-h7ph5\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:42.923710 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.923572    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-d965dfcc-h7ph5\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:42.923710 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.923594    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kserve-provision-location\") pod \"isvc-tensorflow-predictor-d965dfcc-h7ph5\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:42.923710 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.923637    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-proxy-tls\") pod \"isvc-tensorflow-predictor-d965dfcc-h7ph5\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:42.923710 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.923664    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-cabundle-cert\") pod \"isvc-tensorflow-predictor-d965dfcc-h7ph5\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:42.924077 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.924055    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kserve-provision-location\") pod \"isvc-tensorflow-predictor-d965dfcc-h7ph5\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:42.924504 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.924484    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-d965dfcc-h7ph5\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:42.924581 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.924484    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-cabundle-cert\") pod \"isvc-tensorflow-predictor-d965dfcc-h7ph5\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:42.926528 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.926506    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-proxy-tls\") pod \"isvc-tensorflow-predictor-d965dfcc-h7ph5\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:42.986326 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:42.986294    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4zxl\" (UniqueName: \"kubernetes.io/projected/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kube-api-access-t4zxl\") pod \"isvc-tensorflow-predictor-d965dfcc-h7ph5\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:43.088615 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:43.088581    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:43.513990 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:43.513903    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"]
Oct 30 22:42:43.517105 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:42:43.517078    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a0802e_e7a1_4aa0_aa74_fd43db4730cd.slice/crio-58a26959b77d67bcd6afb96fa588a892b1568e11bb499c0b0586f7cd43578444 WatchSource:0}: Error finding container 58a26959b77d67bcd6afb96fa588a892b1568e11bb499c0b0586f7cd43578444: Status 404 returned error can't find the container with id 58a26959b77d67bcd6afb96fa588a892b1568e11bb499c0b0586f7cd43578444
Oct 30 22:42:43.912837 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:43.912797    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" event={"ID":"15a0802e-e7a1-4aa0-aa74-fd43db4730cd","Type":"ContainerStarted","Data":"e7cc7bd93443a5799f70fcd69c105709a2c410441b956178764156db2ed2146f"}
Oct 30 22:42:43.913281 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:43.912869    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" event={"ID":"15a0802e-e7a1-4aa0-aa74-fd43db4730cd","Type":"ContainerStarted","Data":"58a26959b77d67bcd6afb96fa588a892b1568e11bb499c0b0586f7cd43578444"}
Oct 30 22:42:48.934814 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:48.934778    4947 generic.go:356] "Generic (PLEG): container finished" podID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerID="e7cc7bd93443a5799f70fcd69c105709a2c410441b956178764156db2ed2146f" exitCode=0
Oct 30 22:42:48.935214 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:48.934851    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" event={"ID":"15a0802e-e7a1-4aa0-aa74-fd43db4730cd","Type":"ContainerDied","Data":"e7cc7bd93443a5799f70fcd69c105709a2c410441b956178764156db2ed2146f"}
Oct 30 22:42:48.936142 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:48.936124    4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:42:53.959478 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:53.959446    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" event={"ID":"15a0802e-e7a1-4aa0-aa74-fd43db4730cd","Type":"ContainerStarted","Data":"c172de88b18f10cfabb310c25a787eefcfc0071cf13bbc0844cd9a488a3a5f07"}
Oct 30 22:42:53.959870 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:53.959561    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" event={"ID":"15a0802e-e7a1-4aa0-aa74-fd43db4730cd","Type":"ContainerStarted","Data":"1d758dc6fb6630e531a92a3b1c87a70eff08af9af48a7173f94ddcb64133c554"}
Oct 30 22:42:53.959870 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:53.959829    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:53.959974 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:53.959893    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:53.961299 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:53.961275    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 22:42:54.120253 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:54.120066    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" podStartSLOduration=2.3418383609999998 podStartE2EDuration="12.12005048s" totalImagesPullingTime="4.898585727s" totalInitContainerRuntime="4.879626392s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:54.119592915 +0000 UTC m=+2072.118940475" watchObservedRunningTime="2026-10-30 22:42:54.12005048 +0000 UTC m=+2072.119398036"
Oct 30 22:42:54.963863 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:54.963825    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 22:42:59.048199 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:42:59.048156    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:42:59.971149 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:59.970240    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:42:59.971481 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:42:59.971443    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 22:43:09.972424 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:43:09.972388    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:43:22.335520 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:43:22.335415    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:43:22.342172 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:43:22.342152    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:43:29.048627 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:43:29.048581    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:43:33.382007 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:43:33.381922    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"]
Oct 30 22:43:33.382471 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:43:33.382157    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="kserve-container" containerID="cri-o://1d758dc6fb6630e531a92a3b1c87a70eff08af9af48a7173f94ddcb64133c554" gracePeriod=30
Oct 30 22:43:33.382471 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:43:33.382225    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="kube-rbac-proxy" containerID="cri-o://c172de88b18f10cfabb310c25a787eefcfc0071cf13bbc0844cd9a488a3a5f07" gracePeriod=30
Oct 30 22:43:34.122206 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:43:34.122170    4947 generic.go:356] "Generic (PLEG): container finished" podID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerID="c172de88b18f10cfabb310c25a787eefcfc0071cf13bbc0844cd9a488a3a5f07" exitCode=0
Oct 30 22:43:34.122402 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:43:34.122239    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" event={"ID":"15a0802e-e7a1-4aa0-aa74-fd43db4730cd","Type":"ContainerDied","Data":"c172de88b18f10cfabb310c25a787eefcfc0071cf13bbc0844cd9a488a3a5f07"}
Oct 30 22:43:34.964741 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:43:34.964699    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Oct 30 22:43:39.964670 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:43:39.964628    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Oct 30 22:43:44.964586 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:43:44.964545    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Oct 30 22:43:44.965038 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:43:44.964661    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:43:49.964819 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:43:49.964767    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Oct 30 22:43:54.964115 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:43:54.964072    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Oct 30 22:43:59.048436 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:43:59.048392    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:43:59.964715 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:43:59.964676    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Oct 30 22:44:04.020238 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.020211    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:44:04.055339 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.055314    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kube-rbac-proxy-sar-config\") pod \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") "
Oct 30 22:44:04.055472 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.055375    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-proxy-tls\") pod \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") "
Oct 30 22:44:04.055472 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.055430    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kube-api-access-t4zxl\" (UniqueName: \"kubernetes.io/projected/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kube-api-access-t4zxl\") pod \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") "
Oct 30 22:44:04.055472 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.055458    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kserve-provision-location\") pod \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") "
Oct 30 22:44:04.055701 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.055478    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-cabundle-cert\") pod \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\" (UID: \"15a0802e-e7a1-4aa0-aa74-fd43db4730cd\") "
Oct 30 22:44:04.055813 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.055775    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kube-rbac-proxy-sar-config" pod "15a0802e-e7a1-4aa0-aa74-fd43db4730cd" (UID: "15a0802e-e7a1-4aa0-aa74-fd43db4730cd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:04.055926 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.055867    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-cabundle-cert" pod "15a0802e-e7a1-4aa0-aa74-fd43db4730cd" (UID: "15a0802e-e7a1-4aa0-aa74-fd43db4730cd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:04.057708 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.057684    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-proxy-tls" pod "15a0802e-e7a1-4aa0-aa74-fd43db4730cd" (UID: "15a0802e-e7a1-4aa0-aa74-fd43db4730cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:44:04.057808 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.057754    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kube-api-access-t4zxl" pod "15a0802e-e7a1-4aa0-aa74-fd43db4730cd" (UID: "15a0802e-e7a1-4aa0-aa74-fd43db4730cd"). InnerVolumeSpecName "kube-api-access-t4zxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:04.066441 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.066414    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kserve-provision-location" pod "15a0802e-e7a1-4aa0-aa74-fd43db4730cd" (UID: "15a0802e-e7a1-4aa0-aa74-fd43db4730cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:04.155995 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.155918    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:04.155995 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.155944    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4zxl\" (UniqueName: \"kubernetes.io/projected/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kube-api-access-t4zxl\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:04.155995 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.155955    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:04.155995 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.155964    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:04.155995 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.155973    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15a0802e-e7a1-4aa0-aa74-fd43db4730cd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:04.248450 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.248421    4947 generic.go:356] "Generic (PLEG): container finished" podID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerID="1d758dc6fb6630e531a92a3b1c87a70eff08af9af48a7173f94ddcb64133c554" exitCode=137
Oct 30 22:44:04.248626 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.248482    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"
Oct 30 22:44:04.248626 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.248513    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" event={"ID":"15a0802e-e7a1-4aa0-aa74-fd43db4730cd","Type":"ContainerDied","Data":"1d758dc6fb6630e531a92a3b1c87a70eff08af9af48a7173f94ddcb64133c554"}
Oct 30 22:44:04.248626 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.248570    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5" event={"ID":"15a0802e-e7a1-4aa0-aa74-fd43db4730cd","Type":"ContainerDied","Data":"58a26959b77d67bcd6afb96fa588a892b1568e11bb499c0b0586f7cd43578444"}
Oct 30 22:44:04.248626 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.248595    4947 scope.go:122] "RemoveContainer" containerID="c172de88b18f10cfabb310c25a787eefcfc0071cf13bbc0844cd9a488a3a5f07"
Oct 30 22:44:04.256807 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.256787    4947 scope.go:122] "RemoveContainer" containerID="1d758dc6fb6630e531a92a3b1c87a70eff08af9af48a7173f94ddcb64133c554"
Oct 30 22:44:04.263829 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.263813    4947 scope.go:122] "RemoveContainer" containerID="e7cc7bd93443a5799f70fcd69c105709a2c410441b956178764156db2ed2146f"
Oct 30 22:44:04.271064 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.271045    4947 scope.go:122] "RemoveContainer" containerID="c172de88b18f10cfabb310c25a787eefcfc0071cf13bbc0844cd9a488a3a5f07"
Oct 30 22:44:04.271319 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:44:04.271299    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c172de88b18f10cfabb310c25a787eefcfc0071cf13bbc0844cd9a488a3a5f07\": container with ID starting with c172de88b18f10cfabb310c25a787eefcfc0071cf13bbc0844cd9a488a3a5f07 not found: ID does not exist" containerID="c172de88b18f10cfabb310c25a787eefcfc0071cf13bbc0844cd9a488a3a5f07"
Oct 30 22:44:04.271392 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.271328    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c172de88b18f10cfabb310c25a787eefcfc0071cf13bbc0844cd9a488a3a5f07"} err="failed to get container status \"c172de88b18f10cfabb310c25a787eefcfc0071cf13bbc0844cd9a488a3a5f07\": rpc error: code = NotFound desc = could not find container \"c172de88b18f10cfabb310c25a787eefcfc0071cf13bbc0844cd9a488a3a5f07\": container with ID starting with c172de88b18f10cfabb310c25a787eefcfc0071cf13bbc0844cd9a488a3a5f07 not found: ID does not exist"
Oct 30 22:44:04.271392 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.271362    4947 scope.go:122] "RemoveContainer" containerID="1d758dc6fb6630e531a92a3b1c87a70eff08af9af48a7173f94ddcb64133c554"
Oct 30 22:44:04.271618 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:44:04.271600    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d758dc6fb6630e531a92a3b1c87a70eff08af9af48a7173f94ddcb64133c554\": container with ID starting with 1d758dc6fb6630e531a92a3b1c87a70eff08af9af48a7173f94ddcb64133c554 not found: ID does not exist" containerID="1d758dc6fb6630e531a92a3b1c87a70eff08af9af48a7173f94ddcb64133c554"
Oct 30 22:44:04.271659 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.271623    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d758dc6fb6630e531a92a3b1c87a70eff08af9af48a7173f94ddcb64133c554"} err="failed to get container status \"1d758dc6fb6630e531a92a3b1c87a70eff08af9af48a7173f94ddcb64133c554\": rpc error: code = NotFound desc = could not find container \"1d758dc6fb6630e531a92a3b1c87a70eff08af9af48a7173f94ddcb64133c554\": container with ID starting with 1d758dc6fb6630e531a92a3b1c87a70eff08af9af48a7173f94ddcb64133c554 not found: ID does not exist"
Oct 30 22:44:04.271659 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.271639    4947 scope.go:122] "RemoveContainer" containerID="e7cc7bd93443a5799f70fcd69c105709a2c410441b956178764156db2ed2146f"
Oct 30 22:44:04.271854 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:44:04.271836    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7cc7bd93443a5799f70fcd69c105709a2c410441b956178764156db2ed2146f\": container with ID starting with e7cc7bd93443a5799f70fcd69c105709a2c410441b956178764156db2ed2146f not found: ID does not exist" containerID="e7cc7bd93443a5799f70fcd69c105709a2c410441b956178764156db2ed2146f"
Oct 30 22:44:04.271898 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.271861    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7cc7bd93443a5799f70fcd69c105709a2c410441b956178764156db2ed2146f"} err="failed to get container status \"e7cc7bd93443a5799f70fcd69c105709a2c410441b956178764156db2ed2146f\": rpc error: code = NotFound desc = could not find container \"e7cc7bd93443a5799f70fcd69c105709a2c410441b956178764156db2ed2146f\": container with ID starting with e7cc7bd93443a5799f70fcd69c105709a2c410441b956178764156db2ed2146f not found: ID does not exist"
Oct 30 22:44:04.419210 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.419132    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"]
Oct 30 22:44:04.422095 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:04.422064    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-tensorflow-predictor-d965dfcc-h7ph5"]
Oct 30 22:44:06.158756 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.158728    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" path="/var/lib/kubelet/pods/15a0802e-e7a1-4aa0-aa74-fd43db4730cd/volumes"
Oct 30 22:44:06.867558 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.867514    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"]
Oct 30 22:44:06.867926 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.867907    4947 cpu_manager.go:414] "containerMap: removing container" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="kserve-container"
Oct 30 22:44:06.868012 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.867927    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="kserve-container"
Oct 30 22:44:06.868012 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.867954    4947 cpu_manager.go:414] "containerMap: removing container" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="storage-initializer"
Oct 30 22:44:06.868012 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.867963    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="storage-initializer"
Oct 30 22:44:06.868012 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.867987    4947 cpu_manager.go:414] "containerMap: removing container" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="kube-rbac-proxy"
Oct 30 22:44:06.868012 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.867995    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="kube-rbac-proxy"
Oct 30 22:44:06.868258 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.868079    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="kube-rbac-proxy"
Oct 30 22:44:06.868258 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.868094    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a0802e-e7a1-4aa0-aa74-fd43db4730cd" containerName="kserve-container"
Oct 30 22:44:06.872718 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.872697    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:06.975672 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.975637    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Oct 30 22:44:06.975853 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.975704    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:44:06.975853 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.975801    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:44:06.975853 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.975833    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:44:06.975853 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.975838    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:44:06.975853 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.975854    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:44:06.976096 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.975836    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Oct 30 22:44:06.977499 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.977477    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa005be-e967-48ff-8efa-2da7b56f6d44-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:06.977605 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.977502    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"]
Oct 30 22:44:06.977605 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.977519    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa005be-e967-48ff-8efa-2da7b56f6d44-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:06.977605 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.977555    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqdgv\" (UniqueName: \"kubernetes.io/projected/7aa005be-e967-48ff-8efa-2da7b56f6d44-kube-api-access-dqdgv\") pod \"isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:06.977710 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.977609    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa005be-e967-48ff-8efa-2da7b56f6d44-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:06.977710 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:06.977668    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa005be-e967-48ff-8efa-2da7b56f6d44-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:07.078789 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:07.078747    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa005be-e967-48ff-8efa-2da7b56f6d44-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:07.078789 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:07.078794    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa005be-e967-48ff-8efa-2da7b56f6d44-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:07.078978 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:07.078906    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqdgv\" (UniqueName: \"kubernetes.io/projected/7aa005be-e967-48ff-8efa-2da7b56f6d44-kube-api-access-dqdgv\") pod \"isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:07.078978 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:07.078940    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa005be-e967-48ff-8efa-2da7b56f6d44-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:07.079060 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:07.078977    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa005be-e967-48ff-8efa-2da7b56f6d44-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:07.079147 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:07.079128    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa005be-e967-48ff-8efa-2da7b56f6d44-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:07.079529 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:07.079502    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa005be-e967-48ff-8efa-2da7b56f6d44-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:07.079669 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:07.079653    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa005be-e967-48ff-8efa-2da7b56f6d44-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:07.081469 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:07.081450    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa005be-e967-48ff-8efa-2da7b56f6d44-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:07.141543 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:07.141469    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqdgv\" (UniqueName: \"kubernetes.io/projected/7aa005be-e967-48ff-8efa-2da7b56f6d44-kube-api-access-dqdgv\") pod \"isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") " pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:07.189350 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:07.189313    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:07.614225 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:07.614185    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"]
Oct 30 22:44:07.617026 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:44:07.616999    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa005be_e967_48ff_8efa_2da7b56f6d44.slice/crio-7742670508c5004575cbccfa7b6225bb57ae9b3d2a553a5b7c840c1140881f6d WatchSource:0}: Error finding container 7742670508c5004575cbccfa7b6225bb57ae9b3d2a553a5b7c840c1140881f6d: Status 404 returned error can't find the container with id 7742670508c5004575cbccfa7b6225bb57ae9b3d2a553a5b7c840c1140881f6d
Oct 30 22:44:08.271357 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:08.271298    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" event={"ID":"7aa005be-e967-48ff-8efa-2da7b56f6d44","Type":"ContainerStarted","Data":"009ccc9574a100243dbbd5847c31b77c7cf9a6ca5e163c25588b37dc44cbd89d"}
Oct 30 22:44:08.271754 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:08.271395    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" event={"ID":"7aa005be-e967-48ff-8efa-2da7b56f6d44","Type":"ContainerStarted","Data":"7742670508c5004575cbccfa7b6225bb57ae9b3d2a553a5b7c840c1140881f6d"}
Oct 30 22:44:13.290538 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:13.290502    4947 generic.go:356] "Generic (PLEG): container finished" podID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerID="009ccc9574a100243dbbd5847c31b77c7cf9a6ca5e163c25588b37dc44cbd89d" exitCode=0
Oct 30 22:44:13.290955 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:13.290572    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" event={"ID":"7aa005be-e967-48ff-8efa-2da7b56f6d44","Type":"ContainerDied","Data":"009ccc9574a100243dbbd5847c31b77c7cf9a6ca5e163c25588b37dc44cbd89d"}
Oct 30 22:44:14.297539 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:14.297502    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" event={"ID":"7aa005be-e967-48ff-8efa-2da7b56f6d44","Type":"ContainerStarted","Data":"d5c932c21e3a29a7bd12f0e4ebddf25bee192630752e6c6a34b3dfdd12e02131"}
Oct 30 22:44:14.297940 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:14.297566    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" event={"ID":"7aa005be-e967-48ff-8efa-2da7b56f6d44","Type":"ContainerStarted","Data":"735f35bb3068ae6ba7fc87492b8b6987cfb7dc24e3990ef3ff6882ed96cac9da"}
Oct 30 22:44:14.297940 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:14.297907    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:14.298075 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:14.298058    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:14.299089 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:14.299065    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Oct 30 22:44:14.426741 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:14.426687    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" podStartSLOduration=3.318237024 podStartE2EDuration="8.426674133s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.108437109s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:14.424983443 +0000 UTC m=+2152.424331009" watchObservedRunningTime="2026-10-30 22:44:14.426674133 +0000 UTC m=+2152.426021696"
Oct 30 22:44:15.303190 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:15.303143    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Oct 30 22:44:20.307892 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:20.307865    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:20.308373 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:20.308310    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Oct 30 22:44:29.048211 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:44:29.048169    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:44:30.308827 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:30.308795    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:44:47.646131 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:47.646098    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"]
Oct 30 22:44:47.646681 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:47.646365    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="kserve-container" containerID="cri-o://735f35bb3068ae6ba7fc87492b8b6987cfb7dc24e3990ef3ff6882ed96cac9da" gracePeriod=30
Oct 30 22:44:47.646681 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:47.646434    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="kube-rbac-proxy" containerID="cri-o://d5c932c21e3a29a7bd12f0e4ebddf25bee192630752e6c6a34b3dfdd12e02131" gracePeriod=30
Oct 30 22:44:48.444128 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:48.444095    4947 generic.go:356] "Generic (PLEG): container finished" podID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerID="d5c932c21e3a29a7bd12f0e4ebddf25bee192630752e6c6a34b3dfdd12e02131" exitCode=0
Oct 30 22:44:48.444298 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:48.444164    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" event={"ID":"7aa005be-e967-48ff-8efa-2da7b56f6d44","Type":"ContainerDied","Data":"d5c932c21e3a29a7bd12f0e4ebddf25bee192630752e6c6a34b3dfdd12e02131"}
Oct 30 22:44:50.303459 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:50.303421    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Oct 30 22:44:55.304098 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:44:55.304053    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Oct 30 22:44:59.048513 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:44:59.048476    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:45:00.303744 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:00.303700    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Oct 30 22:45:00.304141 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:00.303820    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:45:05.303868 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:05.303772    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Oct 30 22:45:10.304154 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:10.304110    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Oct 30 22:45:15.303976 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:15.303932    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Oct 30 22:45:18.292270 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.292241    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:45:18.351655 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.351623    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa005be-e967-48ff-8efa-2da7b56f6d44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa005be-e967-48ff-8efa-2da7b56f6d44-kube-rbac-proxy-sar-config\") pod \"7aa005be-e967-48ff-8efa-2da7b56f6d44\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") "
Oct 30 22:45:18.351844 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.351669    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aa005be-e967-48ff-8efa-2da7b56f6d44-proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa005be-e967-48ff-8efa-2da7b56f6d44-proxy-tls\") pod \"7aa005be-e967-48ff-8efa-2da7b56f6d44\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") "
Oct 30 22:45:18.351844 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.351715    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa005be-e967-48ff-8efa-2da7b56f6d44-kube-api-access-dqdgv\" (UniqueName: \"kubernetes.io/projected/7aa005be-e967-48ff-8efa-2da7b56f6d44-kube-api-access-dqdgv\") pod \"7aa005be-e967-48ff-8efa-2da7b56f6d44\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") "
Oct 30 22:45:18.351968 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.351885    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa005be-e967-48ff-8efa-2da7b56f6d44-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa005be-e967-48ff-8efa-2da7b56f6d44-kserve-provision-location\") pod \"7aa005be-e967-48ff-8efa-2da7b56f6d44\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") "
Oct 30 22:45:18.351968 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.351933    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa005be-e967-48ff-8efa-2da7b56f6d44-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa005be-e967-48ff-8efa-2da7b56f6d44-cabundle-cert\") pod \"7aa005be-e967-48ff-8efa-2da7b56f6d44\" (UID: \"7aa005be-e967-48ff-8efa-2da7b56f6d44\") "
Oct 30 22:45:18.352115 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.352086    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa005be-e967-48ff-8efa-2da7b56f6d44-kube-rbac-proxy-sar-config" pod "7aa005be-e967-48ff-8efa-2da7b56f6d44" (UID: "7aa005be-e967-48ff-8efa-2da7b56f6d44"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:18.352287 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.352265    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa005be-e967-48ff-8efa-2da7b56f6d44-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.352534 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.352292    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa005be-e967-48ff-8efa-2da7b56f6d44-cabundle-cert" pod "7aa005be-e967-48ff-8efa-2da7b56f6d44" (UID: "7aa005be-e967-48ff-8efa-2da7b56f6d44"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:18.354028 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.354002    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa005be-e967-48ff-8efa-2da7b56f6d44-kube-api-access-dqdgv" pod "7aa005be-e967-48ff-8efa-2da7b56f6d44" (UID: "7aa005be-e967-48ff-8efa-2da7b56f6d44"). InnerVolumeSpecName "kube-api-access-dqdgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:18.354241 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.354222    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa005be-e967-48ff-8efa-2da7b56f6d44-proxy-tls" pod "7aa005be-e967-48ff-8efa-2da7b56f6d44" (UID: "7aa005be-e967-48ff-8efa-2da7b56f6d44"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:18.363416 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.363388    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa005be-e967-48ff-8efa-2da7b56f6d44-kserve-provision-location" pod "7aa005be-e967-48ff-8efa-2da7b56f6d44" (UID: "7aa005be-e967-48ff-8efa-2da7b56f6d44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:18.452818 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.452748    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa005be-e967-48ff-8efa-2da7b56f6d44-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.452818 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.452774    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqdgv\" (UniqueName: \"kubernetes.io/projected/7aa005be-e967-48ff-8efa-2da7b56f6d44-kube-api-access-dqdgv\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.452818 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.452784    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa005be-e967-48ff-8efa-2da7b56f6d44-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.452818 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.452795    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa005be-e967-48ff-8efa-2da7b56f6d44-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.561686 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.561651    4947 generic.go:356] "Generic (PLEG): container finished" podID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerID="735f35bb3068ae6ba7fc87492b8b6987cfb7dc24e3990ef3ff6882ed96cac9da" exitCode=137
Oct 30 22:45:18.561872 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.561713    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"
Oct 30 22:45:18.561872 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.561734    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" event={"ID":"7aa005be-e967-48ff-8efa-2da7b56f6d44","Type":"ContainerDied","Data":"735f35bb3068ae6ba7fc87492b8b6987cfb7dc24e3990ef3ff6882ed96cac9da"}
Oct 30 22:45:18.561872 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.561801    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz" event={"ID":"7aa005be-e967-48ff-8efa-2da7b56f6d44","Type":"ContainerDied","Data":"7742670508c5004575cbccfa7b6225bb57ae9b3d2a553a5b7c840c1140881f6d"}
Oct 30 22:45:18.561872 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.561840    4947 scope.go:122] "RemoveContainer" containerID="d5c932c21e3a29a7bd12f0e4ebddf25bee192630752e6c6a34b3dfdd12e02131"
Oct 30 22:45:18.570568 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.570552    4947 scope.go:122] "RemoveContainer" containerID="735f35bb3068ae6ba7fc87492b8b6987cfb7dc24e3990ef3ff6882ed96cac9da"
Oct 30 22:45:18.578066 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.578048    4947 scope.go:122] "RemoveContainer" containerID="009ccc9574a100243dbbd5847c31b77c7cf9a6ca5e163c25588b37dc44cbd89d"
Oct 30 22:45:18.585093 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.585077    4947 scope.go:122] "RemoveContainer" containerID="d5c932c21e3a29a7bd12f0e4ebddf25bee192630752e6c6a34b3dfdd12e02131"
Oct 30 22:45:18.585380 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:45:18.585362    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c932c21e3a29a7bd12f0e4ebddf25bee192630752e6c6a34b3dfdd12e02131\": container with ID starting with d5c932c21e3a29a7bd12f0e4ebddf25bee192630752e6c6a34b3dfdd12e02131 not found: ID does not exist" containerID="d5c932c21e3a29a7bd12f0e4ebddf25bee192630752e6c6a34b3dfdd12e02131"
Oct 30 22:45:18.585462 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.585392    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c932c21e3a29a7bd12f0e4ebddf25bee192630752e6c6a34b3dfdd12e02131"} err="failed to get container status \"d5c932c21e3a29a7bd12f0e4ebddf25bee192630752e6c6a34b3dfdd12e02131\": rpc error: code = NotFound desc = could not find container \"d5c932c21e3a29a7bd12f0e4ebddf25bee192630752e6c6a34b3dfdd12e02131\": container with ID starting with d5c932c21e3a29a7bd12f0e4ebddf25bee192630752e6c6a34b3dfdd12e02131 not found: ID does not exist"
Oct 30 22:45:18.585462 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.585414    4947 scope.go:122] "RemoveContainer" containerID="735f35bb3068ae6ba7fc87492b8b6987cfb7dc24e3990ef3ff6882ed96cac9da"
Oct 30 22:45:18.585658 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:45:18.585640    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735f35bb3068ae6ba7fc87492b8b6987cfb7dc24e3990ef3ff6882ed96cac9da\": container with ID starting with 735f35bb3068ae6ba7fc87492b8b6987cfb7dc24e3990ef3ff6882ed96cac9da not found: ID does not exist" containerID="735f35bb3068ae6ba7fc87492b8b6987cfb7dc24e3990ef3ff6882ed96cac9da"
Oct 30 22:45:18.585695 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.585664    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735f35bb3068ae6ba7fc87492b8b6987cfb7dc24e3990ef3ff6882ed96cac9da"} err="failed to get container status \"735f35bb3068ae6ba7fc87492b8b6987cfb7dc24e3990ef3ff6882ed96cac9da\": rpc error: code = NotFound desc = could not find container \"735f35bb3068ae6ba7fc87492b8b6987cfb7dc24e3990ef3ff6882ed96cac9da\": container with ID starting with 735f35bb3068ae6ba7fc87492b8b6987cfb7dc24e3990ef3ff6882ed96cac9da not found: ID does not exist"
Oct 30 22:45:18.585695 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.585680    4947 scope.go:122] "RemoveContainer" containerID="009ccc9574a100243dbbd5847c31b77c7cf9a6ca5e163c25588b37dc44cbd89d"
Oct 30 22:45:18.585893 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:45:18.585876    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"009ccc9574a100243dbbd5847c31b77c7cf9a6ca5e163c25588b37dc44cbd89d\": container with ID starting with 009ccc9574a100243dbbd5847c31b77c7cf9a6ca5e163c25588b37dc44cbd89d not found: ID does not exist" containerID="009ccc9574a100243dbbd5847c31b77c7cf9a6ca5e163c25588b37dc44cbd89d"
Oct 30 22:45:18.585945 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.585900    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"009ccc9574a100243dbbd5847c31b77c7cf9a6ca5e163c25588b37dc44cbd89d"} err="failed to get container status \"009ccc9574a100243dbbd5847c31b77c7cf9a6ca5e163c25588b37dc44cbd89d\": rpc error: code = NotFound desc = could not find container \"009ccc9574a100243dbbd5847c31b77c7cf9a6ca5e163c25588b37dc44cbd89d\": container with ID starting with 009ccc9574a100243dbbd5847c31b77c7cf9a6ca5e163c25588b37dc44cbd89d not found: ID does not exist"
Oct 30 22:45:18.732178 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.732100    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"]
Oct 30 22:45:18.740099 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:18.740075    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-tensorflow-runtime-predictor-856fbd56f7-rzqdz"]
Oct 30 22:45:20.157635 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:20.157595    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" path="/var/lib/kubelet/pods/7aa005be-e967-48ff-8efa-2da7b56f6d44/volumes"
Oct 30 22:45:21.070093 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.070056    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"]
Oct 30 22:45:21.070388 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.070375    4947 cpu_manager.go:414] "containerMap: removing container" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="storage-initializer"
Oct 30 22:45:21.070435 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.070389    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="storage-initializer"
Oct 30 22:45:21.070435 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.070399    4947 cpu_manager.go:414] "containerMap: removing container" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="kserve-container"
Oct 30 22:45:21.070435 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.070403    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="kserve-container"
Oct 30 22:45:21.070435 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.070425    4947 cpu_manager.go:414] "containerMap: removing container" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="kube-rbac-proxy"
Oct 30 22:45:21.070435 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.070430    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="kube-rbac-proxy"
Oct 30 22:45:21.070584 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.070484    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="kserve-container"
Oct 30 22:45:21.070584 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.070495    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa005be-e967-48ff-8efa-2da7b56f6d44" containerName="kube-rbac-proxy"
Oct 30 22:45:21.075103 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.075085    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:45:21.129027 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.128999    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:45:21.176268 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.176243    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hdbn\" (UniqueName: \"kubernetes.io/projected/6db5872f-ed36-452f-ac75-775a29eea900-kube-api-access-7hdbn\") pod \"isvc-triton-predictor-798d877db6-krfzd\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") " pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:45:21.176673 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.176278    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db5872f-ed36-452f-ac75-775a29eea900-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-798d877db6-krfzd\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") " pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:45:21.176673 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.176415    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db5872f-ed36-452f-ac75-775a29eea900-proxy-tls\") pod \"isvc-triton-predictor-798d877db6-krfzd\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") " pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:45:21.176673 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.176476    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db5872f-ed36-452f-ac75-775a29eea900-kserve-provision-location\") pod \"isvc-triton-predictor-798d877db6-krfzd\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") " pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:45:21.176673 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.176497    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db5872f-ed36-452f-ac75-775a29eea900-cabundle-cert\") pod \"isvc-triton-predictor-798d877db6-krfzd\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") " pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:45:21.178167 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.178139    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:45:21.178167 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.178143    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Oct 30 22:45:21.178365 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.178190    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:45:21.178365 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.178235    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:45:21.178365 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.178270    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:45:21.178365 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.178349    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-triton-predictor-serving-cert\""
Oct 30 22:45:21.179959 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.179939    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"]
Oct 30 22:45:21.277721 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.277684    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdbn\" (UniqueName: \"kubernetes.io/projected/6db5872f-ed36-452f-ac75-775a29eea900-kube-api-access-7hdbn\") pod \"isvc-triton-predictor-798d877db6-krfzd\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") " pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:45:21.277721 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.277727    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db5872f-ed36-452f-ac75-775a29eea900-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-798d877db6-krfzd\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") " pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:45:21.277973 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.277791    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db5872f-ed36-452f-ac75-775a29eea900-proxy-tls\") pod \"isvc-triton-predictor-798d877db6-krfzd\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") " pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:45:21.277973 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.277826    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db5872f-ed36-452f-ac75-775a29eea900-kserve-provision-location\") pod \"isvc-triton-predictor-798d877db6-krfzd\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") " pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:45:21.277973 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.277851    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db5872f-ed36-452f-ac75-775a29eea900-cabundle-cert\") pod \"isvc-triton-predictor-798d877db6-krfzd\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") " pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:45:21.278418 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.278389    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db5872f-ed36-452f-ac75-775a29eea900-kserve-provision-location\") pod \"isvc-triton-predictor-798d877db6-krfzd\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") " pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:45:21.278565 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.278547    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db5872f-ed36-452f-ac75-775a29eea900-cabundle-cert\") pod \"isvc-triton-predictor-798d877db6-krfzd\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") " pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:45:21.278639 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.278623    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db5872f-ed36-452f-ac75-775a29eea900-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-798d877db6-krfzd\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") " pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:45:21.280447 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.280431    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db5872f-ed36-452f-ac75-775a29eea900-proxy-tls\") pod \"isvc-triton-predictor-798d877db6-krfzd\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") " pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:45:21.340681 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.340622    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hdbn\" (UniqueName: \"kubernetes.io/projected/6db5872f-ed36-452f-ac75-775a29eea900-kube-api-access-7hdbn\") pod \"isvc-triton-predictor-798d877db6-krfzd\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") " pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:45:21.391590 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.391561    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:45:21.820121 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:21.820085    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"]
Oct 30 22:45:21.823671 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:45:21.823640    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db5872f_ed36_452f_ac75_775a29eea900.slice/crio-351e1693e9235048480e345c56b228f964a652c7d5bffba4556576dd0e5185ea WatchSource:0}: Error finding container 351e1693e9235048480e345c56b228f964a652c7d5bffba4556576dd0e5185ea: Status 404 returned error can't find the container with id 351e1693e9235048480e345c56b228f964a652c7d5bffba4556576dd0e5185ea
Oct 30 22:45:22.580883 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:22.580842    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd" event={"ID":"6db5872f-ed36-452f-ac75-775a29eea900","Type":"ContainerStarted","Data":"efa403e47d50dab6ac20d6676c6c1a9d56a5b107472117df298a77d9ce9cd0b9"}
Oct 30 22:45:22.581250 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:22.580906    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd" event={"ID":"6db5872f-ed36-452f-ac75-775a29eea900","Type":"ContainerStarted","Data":"351e1693e9235048480e345c56b228f964a652c7d5bffba4556576dd0e5185ea"}
Oct 30 22:45:26.598209 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:26.598173    4947 generic.go:356] "Generic (PLEG): container finished" podID="6db5872f-ed36-452f-ac75-775a29eea900" containerID="efa403e47d50dab6ac20d6676c6c1a9d56a5b107472117df298a77d9ce9cd0b9" exitCode=0
Oct 30 22:45:26.598602 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:45:26.598249    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd" event={"ID":"6db5872f-ed36-452f-ac75-775a29eea900","Type":"ContainerDied","Data":"efa403e47d50dab6ac20d6676c6c1a9d56a5b107472117df298a77d9ce9cd0b9"}
Oct 30 22:45:29.088598 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:45:29.088515    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:45:59.060875 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:45:59.060809    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:46:29.051916 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:46:29.051872    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:46:59.051047 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:46:59.050962    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:47:22.180911 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:22.180824    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd" event={"ID":"6db5872f-ed36-452f-ac75-775a29eea900","Type":"ContainerStarted","Data":"a1d55c4ada4278ce5b13047a361051d34a8fca3926c75927e13325f534c8b736"}
Oct 30 22:47:22.180911 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:22.180894    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd" event={"ID":"6db5872f-ed36-452f-ac75-775a29eea900","Type":"ContainerStarted","Data":"a6bc36a1d5261ed77e40206fc9d812c0f6ad0d6e379e6c84ac389428557368b4"}
Oct 30 22:47:22.181504 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:22.180931    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:47:22.181504 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:22.180978    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:47:22.182298 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:22.182278    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd" podUID="6db5872f-ed36-452f-ac75-775a29eea900" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Oct 30 22:47:22.316973 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:22.316921    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd" podStartSLOduration=1.779346823 podStartE2EDuration="2m1.316907491s" totalImagesPullingTime="1m55.270744536s" totalInitContainerRuntime="4.266816132s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:22.315871663 +0000 UTC m=+2340.315219226" watchObservedRunningTime="2026-10-30 22:47:22.316907491 +0000 UTC m=+2340.316255054"
Oct 30 22:47:23.185492 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:23.185456    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd" podUID="6db5872f-ed36-452f-ac75-775a29eea900" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Oct 30 22:47:28.190447 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:28.190416    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:47:28.190925 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:28.190898    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd" podUID="6db5872f-ed36-452f-ac75-775a29eea900" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Oct 30 22:47:29.048222 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:47:29.048187    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:47:38.191752 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:38.191726    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:47:43.947490 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:43.947454    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"]
Oct 30 22:47:43.947991 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:43.947694    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd" podUID="6db5872f-ed36-452f-ac75-775a29eea900" containerName="kserve-container" containerID="cri-o://a6bc36a1d5261ed77e40206fc9d812c0f6ad0d6e379e6c84ac389428557368b4" gracePeriod=30
Oct 30 22:47:43.947991 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:43.947797    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd" podUID="6db5872f-ed36-452f-ac75-775a29eea900" containerName="kube-rbac-proxy" containerID="cri-o://a1d55c4ada4278ce5b13047a361051d34a8fca3926c75927e13325f534c8b736" gracePeriod=30
Oct 30 22:47:44.272708 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:44.272629    4947 generic.go:356] "Generic (PLEG): container finished" podID="6db5872f-ed36-452f-ac75-775a29eea900" containerID="a1d55c4ada4278ce5b13047a361051d34a8fca3926c75927e13325f534c8b736" exitCode=0
Oct 30 22:47:44.272851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:44.272706    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd" event={"ID":"6db5872f-ed36-452f-ac75-775a29eea900","Type":"ContainerDied","Data":"a1d55c4ada4278ce5b13047a361051d34a8fca3926c75927e13325f534c8b736"}
Oct 30 22:47:46.723933 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:46.723907    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:47:46.760680 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:46.760631    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db5872f-ed36-452f-ac75-775a29eea900-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db5872f-ed36-452f-ac75-775a29eea900-kserve-provision-location\") pod \"6db5872f-ed36-452f-ac75-775a29eea900\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") "
Oct 30 22:47:46.760680 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:46.760671    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db5872f-ed36-452f-ac75-775a29eea900-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db5872f-ed36-452f-ac75-775a29eea900-kube-rbac-proxy-sar-config\") pod \"6db5872f-ed36-452f-ac75-775a29eea900\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") "
Oct 30 22:47:46.760846 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:46.760709    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db5872f-ed36-452f-ac75-775a29eea900-kube-api-access-7hdbn\" (UniqueName: \"kubernetes.io/projected/6db5872f-ed36-452f-ac75-775a29eea900-kube-api-access-7hdbn\") pod \"6db5872f-ed36-452f-ac75-775a29eea900\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") "
Oct 30 22:47:46.760846 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:46.760778    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db5872f-ed36-452f-ac75-775a29eea900-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db5872f-ed36-452f-ac75-775a29eea900-cabundle-cert\") pod \"6db5872f-ed36-452f-ac75-775a29eea900\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") "
Oct 30 22:47:46.760846 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:46.760817    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db5872f-ed36-452f-ac75-775a29eea900-proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db5872f-ed36-452f-ac75-775a29eea900-proxy-tls\") pod \"6db5872f-ed36-452f-ac75-775a29eea900\" (UID: \"6db5872f-ed36-452f-ac75-775a29eea900\") "
Oct 30 22:47:46.761078 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:46.761054    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db5872f-ed36-452f-ac75-775a29eea900-kube-rbac-proxy-sar-config" pod "6db5872f-ed36-452f-ac75-775a29eea900" (UID: "6db5872f-ed36-452f-ac75-775a29eea900"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:46.761130 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:46.761100    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db5872f-ed36-452f-ac75-775a29eea900-kserve-provision-location" pod "6db5872f-ed36-452f-ac75-775a29eea900" (UID: "6db5872f-ed36-452f-ac75-775a29eea900"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:46.761179 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:46.761155    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db5872f-ed36-452f-ac75-775a29eea900-cabundle-cert" pod "6db5872f-ed36-452f-ac75-775a29eea900" (UID: "6db5872f-ed36-452f-ac75-775a29eea900"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:46.762944 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:46.762915    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db5872f-ed36-452f-ac75-775a29eea900-kube-api-access-7hdbn" pod "6db5872f-ed36-452f-ac75-775a29eea900" (UID: "6db5872f-ed36-452f-ac75-775a29eea900"). InnerVolumeSpecName "kube-api-access-7hdbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:46.763036 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:46.762988    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db5872f-ed36-452f-ac75-775a29eea900-proxy-tls" pod "6db5872f-ed36-452f-ac75-775a29eea900" (UID: "6db5872f-ed36-452f-ac75-775a29eea900"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:46.861518 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:46.861488    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db5872f-ed36-452f-ac75-775a29eea900-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.861518 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:46.861514    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db5872f-ed36-452f-ac75-775a29eea900-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.861518 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:46.861523    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hdbn\" (UniqueName: \"kubernetes.io/projected/6db5872f-ed36-452f-ac75-775a29eea900-kube-api-access-7hdbn\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.861729 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:46.861533    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db5872f-ed36-452f-ac75-775a29eea900-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.861729 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:46.861541    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db5872f-ed36-452f-ac75-775a29eea900-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:47.287568 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:47.287529    4947 generic.go:356] "Generic (PLEG): container finished" podID="6db5872f-ed36-452f-ac75-775a29eea900" containerID="a6bc36a1d5261ed77e40206fc9d812c0f6ad0d6e379e6c84ac389428557368b4" exitCode=0
Oct 30 22:47:47.287731 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:47.287588    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"
Oct 30 22:47:47.287731 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:47.287614    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd" event={"ID":"6db5872f-ed36-452f-ac75-775a29eea900","Type":"ContainerDied","Data":"a6bc36a1d5261ed77e40206fc9d812c0f6ad0d6e379e6c84ac389428557368b4"}
Oct 30 22:47:47.287731 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:47.287674    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd" event={"ID":"6db5872f-ed36-452f-ac75-775a29eea900","Type":"ContainerDied","Data":"351e1693e9235048480e345c56b228f964a652c7d5bffba4556576dd0e5185ea"}
Oct 30 22:47:47.287731 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:47.287702    4947 scope.go:122] "RemoveContainer" containerID="a1d55c4ada4278ce5b13047a361051d34a8fca3926c75927e13325f534c8b736"
Oct 30 22:47:47.296665 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:47.296649    4947 scope.go:122] "RemoveContainer" containerID="a6bc36a1d5261ed77e40206fc9d812c0f6ad0d6e379e6c84ac389428557368b4"
Oct 30 22:47:47.304217 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:47.304201    4947 scope.go:122] "RemoveContainer" containerID="efa403e47d50dab6ac20d6676c6c1a9d56a5b107472117df298a77d9ce9cd0b9"
Oct 30 22:47:47.311384 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:47.311366    4947 scope.go:122] "RemoveContainer" containerID="a1d55c4ada4278ce5b13047a361051d34a8fca3926c75927e13325f534c8b736"
Oct 30 22:47:47.311664 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:47:47.311638    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d55c4ada4278ce5b13047a361051d34a8fca3926c75927e13325f534c8b736\": container with ID starting with a1d55c4ada4278ce5b13047a361051d34a8fca3926c75927e13325f534c8b736 not found: ID does not exist" containerID="a1d55c4ada4278ce5b13047a361051d34a8fca3926c75927e13325f534c8b736"
Oct 30 22:47:47.311710 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:47.311678    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d55c4ada4278ce5b13047a361051d34a8fca3926c75927e13325f534c8b736"} err="failed to get container status \"a1d55c4ada4278ce5b13047a361051d34a8fca3926c75927e13325f534c8b736\": rpc error: code = NotFound desc = could not find container \"a1d55c4ada4278ce5b13047a361051d34a8fca3926c75927e13325f534c8b736\": container with ID starting with a1d55c4ada4278ce5b13047a361051d34a8fca3926c75927e13325f534c8b736 not found: ID does not exist"
Oct 30 22:47:47.311710 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:47.311701    4947 scope.go:122] "RemoveContainer" containerID="a6bc36a1d5261ed77e40206fc9d812c0f6ad0d6e379e6c84ac389428557368b4"
Oct 30 22:47:47.311954 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:47:47.311934    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6bc36a1d5261ed77e40206fc9d812c0f6ad0d6e379e6c84ac389428557368b4\": container with ID starting with a6bc36a1d5261ed77e40206fc9d812c0f6ad0d6e379e6c84ac389428557368b4 not found: ID does not exist" containerID="a6bc36a1d5261ed77e40206fc9d812c0f6ad0d6e379e6c84ac389428557368b4"
Oct 30 22:47:47.311993 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:47.311960    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6bc36a1d5261ed77e40206fc9d812c0f6ad0d6e379e6c84ac389428557368b4"} err="failed to get container status \"a6bc36a1d5261ed77e40206fc9d812c0f6ad0d6e379e6c84ac389428557368b4\": rpc error: code = NotFound desc = could not find container \"a6bc36a1d5261ed77e40206fc9d812c0f6ad0d6e379e6c84ac389428557368b4\": container with ID starting with a6bc36a1d5261ed77e40206fc9d812c0f6ad0d6e379e6c84ac389428557368b4 not found: ID does not exist"
Oct 30 22:47:47.311993 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:47.311974    4947 scope.go:122] "RemoveContainer" containerID="efa403e47d50dab6ac20d6676c6c1a9d56a5b107472117df298a77d9ce9cd0b9"
Oct 30 22:47:47.312190 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:47:47.312168    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efa403e47d50dab6ac20d6676c6c1a9d56a5b107472117df298a77d9ce9cd0b9\": container with ID starting with efa403e47d50dab6ac20d6676c6c1a9d56a5b107472117df298a77d9ce9cd0b9 not found: ID does not exist" containerID="efa403e47d50dab6ac20d6676c6c1a9d56a5b107472117df298a77d9ce9cd0b9"
Oct 30 22:47:47.312228 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:47.312194    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa403e47d50dab6ac20d6676c6c1a9d56a5b107472117df298a77d9ce9cd0b9"} err="failed to get container status \"efa403e47d50dab6ac20d6676c6c1a9d56a5b107472117df298a77d9ce9cd0b9\": rpc error: code = NotFound desc = could not find container \"efa403e47d50dab6ac20d6676c6c1a9d56a5b107472117df298a77d9ce9cd0b9\": container with ID starting with efa403e47d50dab6ac20d6676c6c1a9d56a5b107472117df298a77d9ce9cd0b9 not found: ID does not exist"
Oct 30 22:47:47.456162 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:47.456126    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"]
Oct 30 22:47:47.505729 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:47.505703    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-triton-predictor-798d877db6-krfzd"]
Oct 30 22:47:48.158517 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:48.158482    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db5872f-ed36-452f-ac75-775a29eea900" path="/var/lib/kubelet/pods/6db5872f-ed36-452f-ac75-775a29eea900/volumes"
Oct 30 22:47:50.958794 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:50.958762    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"]
Oct 30 22:47:50.959303 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:50.959120    4947 cpu_manager.go:414] "containerMap: removing container" podUID="6db5872f-ed36-452f-ac75-775a29eea900" containerName="storage-initializer"
Oct 30 22:47:50.959303 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:50.959133    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db5872f-ed36-452f-ac75-775a29eea900" containerName="storage-initializer"
Oct 30 22:47:50.959303 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:50.959141    4947 cpu_manager.go:414] "containerMap: removing container" podUID="6db5872f-ed36-452f-ac75-775a29eea900" containerName="kserve-container"
Oct 30 22:47:50.959303 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:50.959146    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db5872f-ed36-452f-ac75-775a29eea900" containerName="kserve-container"
Oct 30 22:47:50.959303 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:50.959160    4947 cpu_manager.go:414] "containerMap: removing container" podUID="6db5872f-ed36-452f-ac75-775a29eea900" containerName="kube-rbac-proxy"
Oct 30 22:47:50.959303 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:50.959164    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db5872f-ed36-452f-ac75-775a29eea900" containerName="kube-rbac-proxy"
Oct 30 22:47:50.959303 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:50.959225    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db5872f-ed36-452f-ac75-775a29eea900" containerName="kserve-container"
Oct 30 22:47:50.959303 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:50.959234    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db5872f-ed36-452f-ac75-775a29eea900" containerName="kube-rbac-proxy"
Oct 30 22:47:50.976029 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:50.975999    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:47:51.030734 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.030706    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-xgboost-predictor-serving-cert\""
Oct 30 22:47:51.080536 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.080506    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:47:51.080536 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.080523    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:47:51.080717 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.080538    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Oct 30 22:47:51.080717 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.080602    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:47:51.080717 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.080653    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:47:51.080839 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.080766    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:47:51.081993 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.081975    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"]
Oct 30 22:47:51.098709 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.098683    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-proxy-tls\") pod \"isvc-xgboost-predictor-796cb9b9c9-psgv4\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:47:51.098851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.098712    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2tl7\" (UniqueName: \"kubernetes.io/projected/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kube-api-access-p2tl7\") pod \"isvc-xgboost-predictor-796cb9b9c9-psgv4\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:47:51.098851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.098733    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-cabundle-cert\") pod \"isvc-xgboost-predictor-796cb9b9c9-psgv4\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:47:51.098851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.098824    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-796cb9b9c9-psgv4\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:47:51.099029 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.098935    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kserve-provision-location\") pod \"isvc-xgboost-predictor-796cb9b9c9-psgv4\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:47:51.200247 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.200218    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-proxy-tls\") pod \"isvc-xgboost-predictor-796cb9b9c9-psgv4\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:47:51.200425 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.200258    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2tl7\" (UniqueName: \"kubernetes.io/projected/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kube-api-access-p2tl7\") pod \"isvc-xgboost-predictor-796cb9b9c9-psgv4\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:47:51.200425 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.200299    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-cabundle-cert\") pod \"isvc-xgboost-predictor-796cb9b9c9-psgv4\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:47:51.200425 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.200369    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-796cb9b9c9-psgv4\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:47:51.200425 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.200420    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kserve-provision-location\") pod \"isvc-xgboost-predictor-796cb9b9c9-psgv4\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:47:51.200902 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.200879    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kserve-provision-location\") pod \"isvc-xgboost-predictor-796cb9b9c9-psgv4\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:47:51.201098 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.201076    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-cabundle-cert\") pod \"isvc-xgboost-predictor-796cb9b9c9-psgv4\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:47:51.201213 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.201194    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-796cb9b9c9-psgv4\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:47:51.202834 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.202819    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-proxy-tls\") pod \"isvc-xgboost-predictor-796cb9b9c9-psgv4\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:47:51.263592 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.263539    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2tl7\" (UniqueName: \"kubernetes.io/projected/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kube-api-access-p2tl7\") pod \"isvc-xgboost-predictor-796cb9b9c9-psgv4\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:47:51.334601 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.334574    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:47:51.779242 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:51.779207    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"]
Oct 30 22:47:51.782097 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:47:51.782064    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fdbd63_17fe_4d7f_a765_0a0c81f2ef55.slice/crio-f62b84ba9717ca0db15dbb01f1a721903db6aa21a6a69eddb7b604165c42d400 WatchSource:0}: Error finding container f62b84ba9717ca0db15dbb01f1a721903db6aa21a6a69eddb7b604165c42d400: Status 404 returned error can't find the container with id f62b84ba9717ca0db15dbb01f1a721903db6aa21a6a69eddb7b604165c42d400
Oct 30 22:47:52.310547 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:52.310513    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" event={"ID":"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55","Type":"ContainerStarted","Data":"38ab6883110d010e82627ef0582c31f2a21edb72ba02bf9115a755107dfcae1d"}
Oct 30 22:47:52.311032 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:52.310568    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" event={"ID":"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55","Type":"ContainerStarted","Data":"f62b84ba9717ca0db15dbb01f1a721903db6aa21a6a69eddb7b604165c42d400"}
Oct 30 22:47:56.328646 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:56.328617    4947 generic.go:356] "Generic (PLEG): container finished" podID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerID="38ab6883110d010e82627ef0582c31f2a21edb72ba02bf9115a755107dfcae1d" exitCode=0
Oct 30 22:47:56.329043 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:56.328701    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" event={"ID":"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55","Type":"ContainerDied","Data":"38ab6883110d010e82627ef0582c31f2a21edb72ba02bf9115a755107dfcae1d"}
Oct 30 22:47:56.330009 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:47:56.329990    4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:47:59.054067 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:47:59.054012    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:48:07.883063 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:07.882979    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"]
Oct 30 22:48:07.885603 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:07.885581    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:07.939824 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:07.939793    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"openshift-service-ca.crt\""
Oct 30 22:48:07.988888 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:07.988846    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:48:07.989172 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:07.989056    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Oct 30 22:48:07.989172 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:07.989126    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"seaweedfs-s3-creds\""
Oct 30 22:48:07.989172 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:07.989127    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"default-dockercfg-s9cqw\""
Oct 30 22:48:07.989172 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:07.989132    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"kube-root-ca.crt\""
Oct 30 22:48:07.989507 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:07.989199    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Oct 30 22:48:07.991275 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:07.991252    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"]
Oct 30 22:48:08.042538 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:08.042488    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b28ad7b-95ca-447d-affb-20419439eb0d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:08.042720 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:08.042544    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b28ad7b-95ca-447d-affb-20419439eb0d-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:08.042720 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:08.042662    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b28ad7b-95ca-447d-affb-20419439eb0d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:08.042720 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:08.042693    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh7ww\" (UniqueName: \"kubernetes.io/projected/3b28ad7b-95ca-447d-affb-20419439eb0d-kube-api-access-mh7ww\") pod \"isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:08.042920 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:08.042766    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b28ad7b-95ca-447d-affb-20419439eb0d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:08.144056 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:08.143973    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b28ad7b-95ca-447d-affb-20419439eb0d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:08.144056 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:08.144027    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b28ad7b-95ca-447d-affb-20419439eb0d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:08.144056 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:08.144059    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b28ad7b-95ca-447d-affb-20419439eb0d-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:08.144345 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:08.144085    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b28ad7b-95ca-447d-affb-20419439eb0d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:08.144345 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:08.144108    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh7ww\" (UniqueName: \"kubernetes.io/projected/3b28ad7b-95ca-447d-affb-20419439eb0d-kube-api-access-mh7ww\") pod \"isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:08.144621 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:08.144585    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b28ad7b-95ca-447d-affb-20419439eb0d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:08.144827 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:08.144803    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b28ad7b-95ca-447d-affb-20419439eb0d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:08.144904 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:08.144871    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b28ad7b-95ca-447d-affb-20419439eb0d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:08.146964 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:08.146934    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b28ad7b-95ca-447d-affb-20419439eb0d-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:08.207796 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:08.207768    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh7ww\" (UniqueName: \"kubernetes.io/projected/3b28ad7b-95ca-447d-affb-20419439eb0d-kube-api-access-mh7ww\") pod \"isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:08.502326 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:08.502235    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:08.945325 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:08.945281    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"]
Oct 30 22:48:08.949977 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:48:08.949950    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b28ad7b_95ca_447d_affb_20419439eb0d.slice/crio-0f6e5fc3295b779f61ebb268d472a34f2569c41be44b798b491351e05e6e7ca2 WatchSource:0}: Error finding container 0f6e5fc3295b779f61ebb268d472a34f2569c41be44b798b491351e05e6e7ca2: Status 404 returned error can't find the container with id 0f6e5fc3295b779f61ebb268d472a34f2569c41be44b798b491351e05e6e7ca2
Oct 30 22:48:09.390704 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:09.390662    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k" event={"ID":"3b28ad7b-95ca-447d-affb-20419439eb0d","Type":"ContainerStarted","Data":"f5354b6271a6fedcf0b52af2188ea7cf5d83e60d409dc80eb0cdd10c2597886a"}
Oct 30 22:48:09.390891 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:09.390737    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k" event={"ID":"3b28ad7b-95ca-447d-affb-20419439eb0d","Type":"ContainerStarted","Data":"0f6e5fc3295b779f61ebb268d472a34f2569c41be44b798b491351e05e6e7ca2"}
Oct 30 22:48:17.427274 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:17.427235    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" event={"ID":"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55","Type":"ContainerStarted","Data":"a2ad2154c2c72c49b1255773c6042db75f485809823d86ab8f9f829651dc3f5a"}
Oct 30 22:48:17.427729 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:17.427316    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" event={"ID":"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55","Type":"ContainerStarted","Data":"b620d6ac8e6a34f54506846c189e2181e904790ccbfb629a70d1cea3bfbfd9ac"}
Oct 30 22:48:17.427729 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:17.427555    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:48:17.427729 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:17.427596    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:48:17.429085 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:17.429055    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 22:48:17.603460 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:17.603403    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" podStartSLOduration=2.93586812 podStartE2EDuration="27.603387591s" totalImagesPullingTime="20.501515201s" totalInitContainerRuntime="4.16600427s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:17.601908325 +0000 UTC m=+2395.601255899" watchObservedRunningTime="2026-10-30 22:48:17.603387591 +0000 UTC m=+2395.602735189"
Oct 30 22:48:18.431826 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:18.431788    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 22:48:21.450377 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:21.450277    4947 generic.go:356] "Generic (PLEG): container finished" podID="3b28ad7b-95ca-447d-affb-20419439eb0d" containerID="f5354b6271a6fedcf0b52af2188ea7cf5d83e60d409dc80eb0cdd10c2597886a" exitCode=0
Oct 30 22:48:21.450836 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:21.450366    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k" event={"ID":"3b28ad7b-95ca-447d-affb-20419439eb0d","Type":"ContainerDied","Data":"f5354b6271a6fedcf0b52af2188ea7cf5d83e60d409dc80eb0cdd10c2597886a"}
Oct 30 22:48:22.364084 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:22.363980    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:48:22.370186 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:22.370165    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:48:22.457496 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:22.457467    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k" event={"ID":"3b28ad7b-95ca-447d-affb-20419439eb0d","Type":"ContainerStarted","Data":"25eccdcb860ea665b04529fd4f1be68442e416fa51fd121039e7dbea0835ff07"}
Oct 30 22:48:22.457870 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:22.457529    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k" event={"ID":"3b28ad7b-95ca-447d-affb-20419439eb0d","Type":"ContainerStarted","Data":"fa87831262030759e0d9034fdfd6e85f7a26592e3f45d2a914a5696f712a6294"}
Oct 30 22:48:22.457870 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:22.457725    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:22.580825 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:22.580775    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k" podStartSLOduration=3.4069881300000002 podStartE2EDuration="15.580760152s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.173772022s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:22.580020821 +0000 UTC m=+2400.579368411" watchObservedRunningTime="2026-10-30 22:48:22.580760152 +0000 UTC m=+2400.580107715"
Oct 30 22:48:23.443761 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:23.443707    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:48:23.444543 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:23.444500    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 22:48:23.462401 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:23.462368    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:23.463562 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:23.463536    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k" podUID="3b28ad7b-95ca-447d-affb-20419439eb0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Oct 30 22:48:24.467661 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:24.467625    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k" podUID="3b28ad7b-95ca-447d-affb-20419439eb0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Oct 30 22:48:29.048406 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:48:29.048366    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:48:29.472842 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:29.472768    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:29.473357 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:29.473303    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k" podUID="3b28ad7b-95ca-447d-affb-20419439eb0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Oct 30 22:48:33.445407 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:33.445352    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 22:48:39.474102 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:39.474065    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:48:43.445055 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:43.445009    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 22:48:53.445218 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:48:53.445177    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 22:48:59.049639 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:48:59.049605    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:49:03.445297 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:03.445257    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 22:49:03.906533 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:03.906499    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"]
Oct 30 22:49:03.906782 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:03.906739    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k" podUID="3b28ad7b-95ca-447d-affb-20419439eb0d" containerName="kserve-container" containerID="cri-o://fa87831262030759e0d9034fdfd6e85f7a26592e3f45d2a914a5696f712a6294" gracePeriod=30
Oct 30 22:49:03.906861 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:03.906791    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k" podUID="3b28ad7b-95ca-447d-affb-20419439eb0d" containerName="kube-rbac-proxy" containerID="cri-o://25eccdcb860ea665b04529fd4f1be68442e416fa51fd121039e7dbea0835ff07" gracePeriod=30
Oct 30 22:49:04.468678 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:04.468642    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k" podUID="3b28ad7b-95ca-447d-affb-20419439eb0d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Oct 30 22:49:04.644383 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:04.644351    4947 generic.go:356] "Generic (PLEG): container finished" podID="3b28ad7b-95ca-447d-affb-20419439eb0d" containerID="25eccdcb860ea665b04529fd4f1be68442e416fa51fd121039e7dbea0835ff07" exitCode=0
Oct 30 22:49:04.644564 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:04.644363    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k" event={"ID":"3b28ad7b-95ca-447d-affb-20419439eb0d","Type":"ContainerDied","Data":"25eccdcb860ea665b04529fd4f1be68442e416fa51fd121039e7dbea0835ff07"}
Oct 30 22:49:05.844817 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:05.844789    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:49:05.908824 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:05.908796    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b28ad7b-95ca-447d-affb-20419439eb0d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b28ad7b-95ca-447d-affb-20419439eb0d-proxy-tls\") pod \"3b28ad7b-95ca-447d-affb-20419439eb0d\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") "
Oct 30 22:49:05.908962 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:05.908840    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b28ad7b-95ca-447d-affb-20419439eb0d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b28ad7b-95ca-447d-affb-20419439eb0d-kserve-provision-location\") pod \"3b28ad7b-95ca-447d-affb-20419439eb0d\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") "
Oct 30 22:49:05.908962 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:05.908860    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b28ad7b-95ca-447d-affb-20419439eb0d-kube-api-access-mh7ww\" (UniqueName: \"kubernetes.io/projected/3b28ad7b-95ca-447d-affb-20419439eb0d-kube-api-access-mh7ww\") pod \"3b28ad7b-95ca-447d-affb-20419439eb0d\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") "
Oct 30 22:49:05.908962 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:05.908880    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b28ad7b-95ca-447d-affb-20419439eb0d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b28ad7b-95ca-447d-affb-20419439eb0d-cabundle-cert\") pod \"3b28ad7b-95ca-447d-affb-20419439eb0d\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") "
Oct 30 22:49:05.908962 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:05.908908    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b28ad7b-95ca-447d-affb-20419439eb0d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b28ad7b-95ca-447d-affb-20419439eb0d-kube-rbac-proxy-sar-config\") pod \"3b28ad7b-95ca-447d-affb-20419439eb0d\" (UID: \"3b28ad7b-95ca-447d-affb-20419439eb0d\") "
Oct 30 22:49:05.909292 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:05.909268    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b28ad7b-95ca-447d-affb-20419439eb0d-cabundle-cert" pod "3b28ad7b-95ca-447d-affb-20419439eb0d" (UID: "3b28ad7b-95ca-447d-affb-20419439eb0d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:05.909382 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:05.909357    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b28ad7b-95ca-447d-affb-20419439eb0d-kube-rbac-proxy-sar-config" pod "3b28ad7b-95ca-447d-affb-20419439eb0d" (UID: "3b28ad7b-95ca-447d-affb-20419439eb0d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:05.909857 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:05.909835    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b28ad7b-95ca-447d-affb-20419439eb0d-kserve-provision-location" pod "3b28ad7b-95ca-447d-affb-20419439eb0d" (UID: "3b28ad7b-95ca-447d-affb-20419439eb0d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:05.911026 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:05.911002    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b28ad7b-95ca-447d-affb-20419439eb0d-proxy-tls" pod "3b28ad7b-95ca-447d-affb-20419439eb0d" (UID: "3b28ad7b-95ca-447d-affb-20419439eb0d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:05.911163 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:05.911144    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b28ad7b-95ca-447d-affb-20419439eb0d-kube-api-access-mh7ww" pod "3b28ad7b-95ca-447d-affb-20419439eb0d" (UID: "3b28ad7b-95ca-447d-affb-20419439eb0d"). InnerVolumeSpecName "kube-api-access-mh7ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:06.009909 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.009825    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b28ad7b-95ca-447d-affb-20419439eb0d-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:06.009909 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.009857    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b28ad7b-95ca-447d-affb-20419439eb0d-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:06.009909 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.009874    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh7ww\" (UniqueName: \"kubernetes.io/projected/3b28ad7b-95ca-447d-affb-20419439eb0d-kube-api-access-mh7ww\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:06.009909 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.009886    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b28ad7b-95ca-447d-affb-20419439eb0d-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:06.009909 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.009899    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b28ad7b-95ca-447d-affb-20419439eb0d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:06.654576 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.654544    4947 generic.go:356] "Generic (PLEG): container finished" podID="3b28ad7b-95ca-447d-affb-20419439eb0d" containerID="fa87831262030759e0d9034fdfd6e85f7a26592e3f45d2a914a5696f712a6294" exitCode=0
Oct 30 22:49:06.654750 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.654606    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"
Oct 30 22:49:06.654750 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.654619    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k" event={"ID":"3b28ad7b-95ca-447d-affb-20419439eb0d","Type":"ContainerDied","Data":"fa87831262030759e0d9034fdfd6e85f7a26592e3f45d2a914a5696f712a6294"}
Oct 30 22:49:06.654750 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.654679    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k" event={"ID":"3b28ad7b-95ca-447d-affb-20419439eb0d","Type":"ContainerDied","Data":"0f6e5fc3295b779f61ebb268d472a34f2569c41be44b798b491351e05e6e7ca2"}
Oct 30 22:49:06.654750 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.654706    4947 scope.go:122] "RemoveContainer" containerID="25eccdcb860ea665b04529fd4f1be68442e416fa51fd121039e7dbea0835ff07"
Oct 30 22:49:06.662740 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.662715    4947 scope.go:122] "RemoveContainer" containerID="fa87831262030759e0d9034fdfd6e85f7a26592e3f45d2a914a5696f712a6294"
Oct 30 22:49:06.669761 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.669737    4947 scope.go:122] "RemoveContainer" containerID="f5354b6271a6fedcf0b52af2188ea7cf5d83e60d409dc80eb0cdd10c2597886a"
Oct 30 22:49:06.676871 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.676853    4947 scope.go:122] "RemoveContainer" containerID="25eccdcb860ea665b04529fd4f1be68442e416fa51fd121039e7dbea0835ff07"
Oct 30 22:49:06.677114 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:49:06.677095    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25eccdcb860ea665b04529fd4f1be68442e416fa51fd121039e7dbea0835ff07\": container with ID starting with 25eccdcb860ea665b04529fd4f1be68442e416fa51fd121039e7dbea0835ff07 not found: ID does not exist" containerID="25eccdcb860ea665b04529fd4f1be68442e416fa51fd121039e7dbea0835ff07"
Oct 30 22:49:06.677165 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.677124    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25eccdcb860ea665b04529fd4f1be68442e416fa51fd121039e7dbea0835ff07"} err="failed to get container status \"25eccdcb860ea665b04529fd4f1be68442e416fa51fd121039e7dbea0835ff07\": rpc error: code = NotFound desc = could not find container \"25eccdcb860ea665b04529fd4f1be68442e416fa51fd121039e7dbea0835ff07\": container with ID starting with 25eccdcb860ea665b04529fd4f1be68442e416fa51fd121039e7dbea0835ff07 not found: ID does not exist"
Oct 30 22:49:06.677165 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.677141    4947 scope.go:122] "RemoveContainer" containerID="fa87831262030759e0d9034fdfd6e85f7a26592e3f45d2a914a5696f712a6294"
Oct 30 22:49:06.677409 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:49:06.677395    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa87831262030759e0d9034fdfd6e85f7a26592e3f45d2a914a5696f712a6294\": container with ID starting with fa87831262030759e0d9034fdfd6e85f7a26592e3f45d2a914a5696f712a6294 not found: ID does not exist" containerID="fa87831262030759e0d9034fdfd6e85f7a26592e3f45d2a914a5696f712a6294"
Oct 30 22:49:06.677460 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.677414    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa87831262030759e0d9034fdfd6e85f7a26592e3f45d2a914a5696f712a6294"} err="failed to get container status \"fa87831262030759e0d9034fdfd6e85f7a26592e3f45d2a914a5696f712a6294\": rpc error: code = NotFound desc = could not find container \"fa87831262030759e0d9034fdfd6e85f7a26592e3f45d2a914a5696f712a6294\": container with ID starting with fa87831262030759e0d9034fdfd6e85f7a26592e3f45d2a914a5696f712a6294 not found: ID does not exist"
Oct 30 22:49:06.677460 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.677427    4947 scope.go:122] "RemoveContainer" containerID="f5354b6271a6fedcf0b52af2188ea7cf5d83e60d409dc80eb0cdd10c2597886a"
Oct 30 22:49:06.677646 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:49:06.677627    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5354b6271a6fedcf0b52af2188ea7cf5d83e60d409dc80eb0cdd10c2597886a\": container with ID starting with f5354b6271a6fedcf0b52af2188ea7cf5d83e60d409dc80eb0cdd10c2597886a not found: ID does not exist" containerID="f5354b6271a6fedcf0b52af2188ea7cf5d83e60d409dc80eb0cdd10c2597886a"
Oct 30 22:49:06.677714 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.677654    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5354b6271a6fedcf0b52af2188ea7cf5d83e60d409dc80eb0cdd10c2597886a"} err="failed to get container status \"f5354b6271a6fedcf0b52af2188ea7cf5d83e60d409dc80eb0cdd10c2597886a\": rpc error: code = NotFound desc = could not find container \"f5354b6271a6fedcf0b52af2188ea7cf5d83e60d409dc80eb0cdd10c2597886a\": container with ID starting with f5354b6271a6fedcf0b52af2188ea7cf5d83e60d409dc80eb0cdd10c2597886a not found: ID does not exist"
Oct 30 22:49:06.866618 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.866583    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"]
Oct 30 22:49:06.916548 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:06.916472    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-predictor-7bdfdc964d-qfx2k"]
Oct 30 22:49:08.158300 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:08.158271    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b28ad7b-95ca-447d-affb-20419439eb0d" path="/var/lib/kubelet/pods/3b28ad7b-95ca-447d-affb-20419439eb0d/volumes"
Oct 30 22:49:09.034946 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.034914    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"]
Oct 30 22:49:09.035256 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.035242    4947 cpu_manager.go:414] "containerMap: removing container" podUID="3b28ad7b-95ca-447d-affb-20419439eb0d" containerName="kserve-container"
Oct 30 22:49:09.035303 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.035256    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b28ad7b-95ca-447d-affb-20419439eb0d" containerName="kserve-container"
Oct 30 22:49:09.035303 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.035266    4947 cpu_manager.go:414] "containerMap: removing container" podUID="3b28ad7b-95ca-447d-affb-20419439eb0d" containerName="storage-initializer"
Oct 30 22:49:09.035303 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.035272    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b28ad7b-95ca-447d-affb-20419439eb0d" containerName="storage-initializer"
Oct 30 22:49:09.035303 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.035286    4947 cpu_manager.go:414] "containerMap: removing container" podUID="3b28ad7b-95ca-447d-affb-20419439eb0d" containerName="kube-rbac-proxy"
Oct 30 22:49:09.035303 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.035291    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b28ad7b-95ca-447d-affb-20419439eb0d" containerName="kube-rbac-proxy"
Oct 30 22:49:09.035496 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.035365    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b28ad7b-95ca-447d-affb-20419439eb0d" containerName="kserve-container"
Oct 30 22:49:09.035496 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.035375    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b28ad7b-95ca-447d-affb-20419439eb0d" containerName="kube-rbac-proxy"
Oct 30 22:49:09.038307 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.038291    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:09.135946 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.135912    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:09.136099 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.135966    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:09.136099 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.136028    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:09.136099 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.136055    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj55w\" (UniqueName: \"kubernetes.io/projected/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kube-api-access-nj55w\") pod \"isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:09.136099 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.136069    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:09.141765 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.141739    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"openshift-service-ca.crt\""
Oct 30 22:49:09.141765 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.141750    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Oct 30 22:49:09.141963 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.141753    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"kube-root-ca.crt\""
Oct 30 22:49:09.141963 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.141804    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"seaweedfs-s3-creds\""
Oct 30 22:49:09.141963 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.141841    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:49:09.141963 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.141866    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Oct 30 22:49:09.141963 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.141952    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"default-dockercfg-s9cqw\""
Oct 30 22:49:09.143773 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.143753    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"]
Oct 30 22:49:09.237299 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.237269    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:09.237755 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.237358    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:09.237755 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.237407    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:09.237755 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.237453    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj55w\" (UniqueName: \"kubernetes.io/projected/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kube-api-access-nj55w\") pod \"isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:09.237755 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.237482    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:09.237944 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.237852    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:09.238132 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.238110    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:09.238173 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.238109    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:09.239969 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.239953    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:09.300690 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.300662    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj55w\" (UniqueName: \"kubernetes.io/projected/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kube-api-access-nj55w\") pod \"isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") " pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:09.351314 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.351288    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:09.776156 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:09.775919    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"]
Oct 30 22:49:09.778750 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:49:09.778721    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0da5b5_e0f3_44ee_a47e_7b67f78b2248.slice/crio-85d954d92b23a35aed6fe47a51e63a4a1eea4cfc3446325b720943a4e2494e74 WatchSource:0}: Error finding container 85d954d92b23a35aed6fe47a51e63a4a1eea4cfc3446325b720943a4e2494e74: Status 404 returned error can't find the container with id 85d954d92b23a35aed6fe47a51e63a4a1eea4cfc3446325b720943a4e2494e74
Oct 30 22:49:10.672569 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:10.672532    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg" event={"ID":"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248","Type":"ContainerStarted","Data":"af437a8f80bc37295ec3b64685ef7edd8918cb05c4f0f447ec032f37afa4cbfa"}
Oct 30 22:49:10.673009 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:10.672591    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg" event={"ID":"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248","Type":"ContainerStarted","Data":"85d954d92b23a35aed6fe47a51e63a4a1eea4cfc3446325b720943a4e2494e74"}
Oct 30 22:49:13.444997 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:13.444955    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 22:49:16.699630 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:16.699593    4947 generic.go:356] "Generic (PLEG): container finished" podID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" containerID="af437a8f80bc37295ec3b64685ef7edd8918cb05c4f0f447ec032f37afa4cbfa" exitCode=0
Oct 30 22:49:16.699630 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:16.699634    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg" event={"ID":"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248","Type":"ContainerDied","Data":"af437a8f80bc37295ec3b64685ef7edd8918cb05c4f0f447ec032f37afa4cbfa"}
Oct 30 22:49:17.707152 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:17.707118    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg" event={"ID":"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248","Type":"ContainerStarted","Data":"462c611e2444b8e2375406c4d2120a703b725c81aee6086c43aa675020ff2737"}
Oct 30 22:49:17.707574 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:17.707210    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg" event={"ID":"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248","Type":"ContainerStarted","Data":"bd2224cd0c3824792c06654bcc79aecb47b519ce68c9b12f7adcc1051d01e8cf"}
Oct 30 22:49:17.707574 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:17.707416    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:17.831627 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:17.831577    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg" podStartSLOduration=3.4597419719999998 podStartE2EDuration="9.83156381s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.371821838s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:17.830431578 +0000 UTC m=+2455.829779141" watchObservedRunningTime="2026-10-30 22:49:17.83156381 +0000 UTC m=+2455.830911372"
Oct 30 22:49:18.712484 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:18.712456    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:18.713665 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:18.713637    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg" podUID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Oct 30 22:49:19.717885 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:19.717846    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg" podUID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Oct 30 22:49:23.444913 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:23.444874    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 22:49:24.722888 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:24.722855    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:24.723477 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:24.723444    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg" podUID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Oct 30 22:49:29.047990 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:49:29.047953    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:49:33.445437 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:33.445358    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:49:34.724534 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:34.724503    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:41.481241 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:41.481206    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"]
Oct 30 22:49:41.481814 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:41.481451    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kserve-container" containerID="cri-o://b620d6ac8e6a34f54506846c189e2181e904790ccbfb629a70d1cea3bfbfd9ac" gracePeriod=30
Oct 30 22:49:41.481814 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:41.481481    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kube-rbac-proxy" containerID="cri-o://a2ad2154c2c72c49b1255773c6042db75f485809823d86ab8f9f829651dc3f5a" gracePeriod=30
Oct 30 22:49:41.821461 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:41.821426    4947 generic.go:356] "Generic (PLEG): container finished" podID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerID="a2ad2154c2c72c49b1255773c6042db75f485809823d86ab8f9f829651dc3f5a" exitCode=0
Oct 30 22:49:41.821631 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:41.821458    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" event={"ID":"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55","Type":"ContainerDied","Data":"a2ad2154c2c72c49b1255773c6042db75f485809823d86ab8f9f829651dc3f5a"}
Oct 30 22:49:43.432867 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:43.432824    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Oct 30 22:49:43.444759 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:43.444730    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 22:49:45.629099 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.629068    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:49:45.735307 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.735222    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-cabundle-cert\") pod \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") "
Oct 30 22:49:45.735307 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.735303    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kube-rbac-proxy-sar-config\") pod \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") "
Oct 30 22:49:45.735555 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.735354    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kserve-provision-location\") pod \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") "
Oct 30 22:49:45.735555 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.735416    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-proxy-tls\" (UniqueName: \"kubernetes.io/secret/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-proxy-tls\") pod \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") "
Oct 30 22:49:45.735555 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.735454    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kube-api-access-p2tl7\" (UniqueName: \"kubernetes.io/projected/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kube-api-access-p2tl7\") pod \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\" (UID: \"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55\") "
Oct 30 22:49:45.735694 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.735618    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-cabundle-cert" pod "50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" (UID: "50fdbd63-17fe-4d7f-a765-0a0c81f2ef55"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:45.735738 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.735708    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kserve-provision-location" pod "50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" (UID: "50fdbd63-17fe-4d7f-a765-0a0c81f2ef55"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:45.735738 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.735723    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:45.735801 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.735725    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kube-rbac-proxy-sar-config" pod "50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" (UID: "50fdbd63-17fe-4d7f-a765-0a0c81f2ef55"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:45.737671 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.737637    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kube-api-access-p2tl7" pod "50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" (UID: "50fdbd63-17fe-4d7f-a765-0a0c81f2ef55"). InnerVolumeSpecName "kube-api-access-p2tl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:45.737782 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.737711    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-proxy-tls" pod "50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" (UID: "50fdbd63-17fe-4d7f-a765-0a0c81f2ef55"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:45.836598 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.836562    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:45.836598 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.836599    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2tl7\" (UniqueName: \"kubernetes.io/projected/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kube-api-access-p2tl7\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:45.836843 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.836617    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:45.836843 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.836633    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:45.838369 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.838313    4947 generic.go:356] "Generic (PLEG): container finished" podID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerID="b620d6ac8e6a34f54506846c189e2181e904790ccbfb629a70d1cea3bfbfd9ac" exitCode=0
Oct 30 22:49:45.838473 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.838377    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" event={"ID":"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55","Type":"ContainerDied","Data":"b620d6ac8e6a34f54506846c189e2181e904790ccbfb629a70d1cea3bfbfd9ac"}
Oct 30 22:49:45.838473 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.838402    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"
Oct 30 22:49:45.838473 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.838439    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4" event={"ID":"50fdbd63-17fe-4d7f-a765-0a0c81f2ef55","Type":"ContainerDied","Data":"f62b84ba9717ca0db15dbb01f1a721903db6aa21a6a69eddb7b604165c42d400"}
Oct 30 22:49:45.838473 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.838466    4947 scope.go:122] "RemoveContainer" containerID="a2ad2154c2c72c49b1255773c6042db75f485809823d86ab8f9f829651dc3f5a"
Oct 30 22:49:45.847820 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.847805    4947 scope.go:122] "RemoveContainer" containerID="b620d6ac8e6a34f54506846c189e2181e904790ccbfb629a70d1cea3bfbfd9ac"
Oct 30 22:49:45.855118 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.855099    4947 scope.go:122] "RemoveContainer" containerID="38ab6883110d010e82627ef0582c31f2a21edb72ba02bf9115a755107dfcae1d"
Oct 30 22:49:45.863876 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.863855    4947 scope.go:122] "RemoveContainer" containerID="a2ad2154c2c72c49b1255773c6042db75f485809823d86ab8f9f829651dc3f5a"
Oct 30 22:49:45.864165 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:49:45.864143    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ad2154c2c72c49b1255773c6042db75f485809823d86ab8f9f829651dc3f5a\": container with ID starting with a2ad2154c2c72c49b1255773c6042db75f485809823d86ab8f9f829651dc3f5a not found: ID does not exist" containerID="a2ad2154c2c72c49b1255773c6042db75f485809823d86ab8f9f829651dc3f5a"
Oct 30 22:49:45.864237 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.864173    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ad2154c2c72c49b1255773c6042db75f485809823d86ab8f9f829651dc3f5a"} err="failed to get container status \"a2ad2154c2c72c49b1255773c6042db75f485809823d86ab8f9f829651dc3f5a\": rpc error: code = NotFound desc = could not find container \"a2ad2154c2c72c49b1255773c6042db75f485809823d86ab8f9f829651dc3f5a\": container with ID starting with a2ad2154c2c72c49b1255773c6042db75f485809823d86ab8f9f829651dc3f5a not found: ID does not exist"
Oct 30 22:49:45.864237 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.864191    4947 scope.go:122] "RemoveContainer" containerID="b620d6ac8e6a34f54506846c189e2181e904790ccbfb629a70d1cea3bfbfd9ac"
Oct 30 22:49:45.864470 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:49:45.864454    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b620d6ac8e6a34f54506846c189e2181e904790ccbfb629a70d1cea3bfbfd9ac\": container with ID starting with b620d6ac8e6a34f54506846c189e2181e904790ccbfb629a70d1cea3bfbfd9ac not found: ID does not exist" containerID="b620d6ac8e6a34f54506846c189e2181e904790ccbfb629a70d1cea3bfbfd9ac"
Oct 30 22:49:45.864522 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.864475    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b620d6ac8e6a34f54506846c189e2181e904790ccbfb629a70d1cea3bfbfd9ac"} err="failed to get container status \"b620d6ac8e6a34f54506846c189e2181e904790ccbfb629a70d1cea3bfbfd9ac\": rpc error: code = NotFound desc = could not find container \"b620d6ac8e6a34f54506846c189e2181e904790ccbfb629a70d1cea3bfbfd9ac\": container with ID starting with b620d6ac8e6a34f54506846c189e2181e904790ccbfb629a70d1cea3bfbfd9ac not found: ID does not exist"
Oct 30 22:49:45.864522 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.864488    4947 scope.go:122] "RemoveContainer" containerID="38ab6883110d010e82627ef0582c31f2a21edb72ba02bf9115a755107dfcae1d"
Oct 30 22:49:45.864714 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:49:45.864699    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ab6883110d010e82627ef0582c31f2a21edb72ba02bf9115a755107dfcae1d\": container with ID starting with 38ab6883110d010e82627ef0582c31f2a21edb72ba02bf9115a755107dfcae1d not found: ID does not exist" containerID="38ab6883110d010e82627ef0582c31f2a21edb72ba02bf9115a755107dfcae1d"
Oct 30 22:49:45.864754 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:45.864716    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ab6883110d010e82627ef0582c31f2a21edb72ba02bf9115a755107dfcae1d"} err="failed to get container status \"38ab6883110d010e82627ef0582c31f2a21edb72ba02bf9115a755107dfcae1d\": rpc error: code = NotFound desc = could not find container \"38ab6883110d010e82627ef0582c31f2a21edb72ba02bf9115a755107dfcae1d\": container with ID starting with 38ab6883110d010e82627ef0582c31f2a21edb72ba02bf9115a755107dfcae1d not found: ID does not exist"
Oct 30 22:49:46.006501 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:46.006417    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"]
Oct 30 22:49:46.011939 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:46.011914    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-predictor-796cb9b9c9-psgv4"]
Oct 30 22:49:46.158424 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:46.158385    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" path="/var/lib/kubelet/pods/50fdbd63-17fe-4d7f-a765-0a0c81f2ef55/volumes"
Oct 30 22:49:48.556618 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.556585    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"]
Oct 30 22:49:48.556983 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.556907    4947 cpu_manager.go:414] "containerMap: removing container" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kube-rbac-proxy"
Oct 30 22:49:48.556983 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.556918    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kube-rbac-proxy"
Oct 30 22:49:48.556983 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.556929    4947 cpu_manager.go:414] "containerMap: removing container" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="storage-initializer"
Oct 30 22:49:48.556983 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.556933    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="storage-initializer"
Oct 30 22:49:48.556983 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.556949    4947 cpu_manager.go:414] "containerMap: removing container" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kserve-container"
Oct 30 22:49:48.556983 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.556954    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kserve-container"
Oct 30 22:49:48.557167 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.556999    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kube-rbac-proxy"
Oct 30 22:49:48.557167 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.557008    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="50fdbd63-17fe-4d7f-a765-0a0c81f2ef55" containerName="kserve-container"
Oct 30 22:49:48.559934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.559917    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:48.656682 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.656651    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btdw4\" (UniqueName: \"kubernetes.io/projected/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kube-api-access-btdw4\") pod \"isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:48.656856 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.656689    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:48.656856 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.656710    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:48.656856 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.656826    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:48.656984 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.656856    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:48.663985 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.663934    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 30 22:49:48.664137 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.664051    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:49:48.664137 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.663963    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:49:48.664256 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.664160    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:49:48.664256 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.664220    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:49:48.664431 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.664403    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Oct 30 22:49:48.665402 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.665365    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:49:48.665839 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.665815    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"]
Oct 30 22:49:48.757564 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.757481    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdw4\" (UniqueName: \"kubernetes.io/projected/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kube-api-access-btdw4\") pod \"isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:48.757564 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.757524    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:48.757564 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.757553    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:48.757868 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.757612    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:48.757868 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.757638    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:48.758191 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.758163    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:48.758279 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.758252    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:48.758392 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.758374    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:48.760197 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.760176    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:48.820767 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.820735    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btdw4\" (UniqueName: \"kubernetes.io/projected/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kube-api-access-btdw4\") pod \"isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:48.876245 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:48.876210    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:49.301747 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:49.301575    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"]
Oct 30 22:49:49.304562 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:49:49.304536    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbba0f25_356f_451a_89c2_f3a2e84c0ba2.slice/crio-d5a7068d1111b4d5d2e85727469a7f5ab9ad2ee35222c4603c5fd7a80ea8197e WatchSource:0}: Error finding container d5a7068d1111b4d5d2e85727469a7f5ab9ad2ee35222c4603c5fd7a80ea8197e: Status 404 returned error can't find the container with id d5a7068d1111b4d5d2e85727469a7f5ab9ad2ee35222c4603c5fd7a80ea8197e
Oct 30 22:49:49.673287 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:49.673212    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"]
Oct 30 22:49:49.673672 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:49.673530    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg" podUID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" containerName="kserve-container" containerID="cri-o://bd2224cd0c3824792c06654bcc79aecb47b519ce68c9b12f7adcc1051d01e8cf" gracePeriod=30
Oct 30 22:49:49.673672 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:49.673600    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg" podUID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" containerName="kube-rbac-proxy" containerID="cri-o://462c611e2444b8e2375406c4d2120a703b725c81aee6086c43aa675020ff2737" gracePeriod=30
Oct 30 22:49:49.717969 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:49.717938    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg" podUID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Oct 30 22:49:49.858311 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:49.858275    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq" event={"ID":"fbba0f25-356f-451a-89c2-f3a2e84c0ba2","Type":"ContainerStarted","Data":"2373c2d9f34dc41acf98dcb04a4e578cc07640458ab77f0bec88ac847c908639"}
Oct 30 22:49:49.858579 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:49.858366    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq" event={"ID":"fbba0f25-356f-451a-89c2-f3a2e84c0ba2","Type":"ContainerStarted","Data":"d5a7068d1111b4d5d2e85727469a7f5ab9ad2ee35222c4603c5fd7a80ea8197e"}
Oct 30 22:49:49.860837 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:49.860807    4947 generic.go:356] "Generic (PLEG): container finished" podID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" containerID="462c611e2444b8e2375406c4d2120a703b725c81aee6086c43aa675020ff2737" exitCode=0
Oct 30 22:49:49.860939 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:49.860863    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg" event={"ID":"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248","Type":"ContainerDied","Data":"462c611e2444b8e2375406c4d2120a703b725c81aee6086c43aa675020ff2737"}
Oct 30 22:49:51.024435 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.024402    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:51.073989 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.073962    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-cabundle-cert\") pod \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") "
Oct 30 22:49:51.074130 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.073999    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kube-rbac-proxy-sar-config\") pod \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") "
Oct 30 22:49:51.074130 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.074020    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-proxy-tls\") pod \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") "
Oct 30 22:49:51.074255 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.074152    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kube-api-access-nj55w\" (UniqueName: \"kubernetes.io/projected/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kube-api-access-nj55w\") pod \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") "
Oct 30 22:49:51.074255 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.074214    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kserve-provision-location\") pod \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\" (UID: \"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248\") "
Oct 30 22:49:51.074393 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.074359    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kube-rbac-proxy-sar-config" pod "5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" (UID: "5c0da5b5-e0f3-44ee-a47e-7b67f78b2248"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:51.074458 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.074385    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-cabundle-cert" pod "5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" (UID: "5c0da5b5-e0f3-44ee-a47e-7b67f78b2248"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:51.074579 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.074557    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:51.074646 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.074584    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:51.075070 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.075051    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kserve-provision-location" pod "5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" (UID: "5c0da5b5-e0f3-44ee-a47e-7b67f78b2248"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:51.076248 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.076229    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-proxy-tls" pod "5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" (UID: "5c0da5b5-e0f3-44ee-a47e-7b67f78b2248"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:51.076370 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.076279    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kube-api-access-nj55w" pod "5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" (UID: "5c0da5b5-e0f3-44ee-a47e-7b67f78b2248"). InnerVolumeSpecName "kube-api-access-nj55w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:51.175111 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.175034    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:51.175111 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.175062    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj55w\" (UniqueName: \"kubernetes.io/projected/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kube-api-access-nj55w\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:51.175111 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.175073    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:51.872710 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.872671    4947 generic.go:356] "Generic (PLEG): container finished" podID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" containerID="bd2224cd0c3824792c06654bcc79aecb47b519ce68c9b12f7adcc1051d01e8cf" exitCode=0
Oct 30 22:49:51.872922 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.872741    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"
Oct 30 22:49:51.872922 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.872762    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg" event={"ID":"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248","Type":"ContainerDied","Data":"bd2224cd0c3824792c06654bcc79aecb47b519ce68c9b12f7adcc1051d01e8cf"}
Oct 30 22:49:51.872922 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.872832    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg" event={"ID":"5c0da5b5-e0f3-44ee-a47e-7b67f78b2248","Type":"ContainerDied","Data":"85d954d92b23a35aed6fe47a51e63a4a1eea4cfc3446325b720943a4e2494e74"}
Oct 30 22:49:51.872922 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.872860    4947 scope.go:122] "RemoveContainer" containerID="462c611e2444b8e2375406c4d2120a703b725c81aee6086c43aa675020ff2737"
Oct 30 22:49:51.881537 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.881519    4947 scope.go:122] "RemoveContainer" containerID="bd2224cd0c3824792c06654bcc79aecb47b519ce68c9b12f7adcc1051d01e8cf"
Oct 30 22:49:51.889005 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.888854    4947 scope.go:122] "RemoveContainer" containerID="af437a8f80bc37295ec3b64685ef7edd8918cb05c4f0f447ec032f37afa4cbfa"
Oct 30 22:49:51.896262 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.896248    4947 scope.go:122] "RemoveContainer" containerID="462c611e2444b8e2375406c4d2120a703b725c81aee6086c43aa675020ff2737"
Oct 30 22:49:51.896522 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:49:51.896505    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462c611e2444b8e2375406c4d2120a703b725c81aee6086c43aa675020ff2737\": container with ID starting with 462c611e2444b8e2375406c4d2120a703b725c81aee6086c43aa675020ff2737 not found: ID does not exist" containerID="462c611e2444b8e2375406c4d2120a703b725c81aee6086c43aa675020ff2737"
Oct 30 22:49:51.896581 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.896535    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462c611e2444b8e2375406c4d2120a703b725c81aee6086c43aa675020ff2737"} err="failed to get container status \"462c611e2444b8e2375406c4d2120a703b725c81aee6086c43aa675020ff2737\": rpc error: code = NotFound desc = could not find container \"462c611e2444b8e2375406c4d2120a703b725c81aee6086c43aa675020ff2737\": container with ID starting with 462c611e2444b8e2375406c4d2120a703b725c81aee6086c43aa675020ff2737 not found: ID does not exist"
Oct 30 22:49:51.896581 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.896551    4947 scope.go:122] "RemoveContainer" containerID="bd2224cd0c3824792c06654bcc79aecb47b519ce68c9b12f7adcc1051d01e8cf"
Oct 30 22:49:51.896760 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:49:51.896744    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd2224cd0c3824792c06654bcc79aecb47b519ce68c9b12f7adcc1051d01e8cf\": container with ID starting with bd2224cd0c3824792c06654bcc79aecb47b519ce68c9b12f7adcc1051d01e8cf not found: ID does not exist" containerID="bd2224cd0c3824792c06654bcc79aecb47b519ce68c9b12f7adcc1051d01e8cf"
Oct 30 22:49:51.896792 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.896773    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd2224cd0c3824792c06654bcc79aecb47b519ce68c9b12f7adcc1051d01e8cf"} err="failed to get container status \"bd2224cd0c3824792c06654bcc79aecb47b519ce68c9b12f7adcc1051d01e8cf\": rpc error: code = NotFound desc = could not find container \"bd2224cd0c3824792c06654bcc79aecb47b519ce68c9b12f7adcc1051d01e8cf\": container with ID starting with bd2224cd0c3824792c06654bcc79aecb47b519ce68c9b12f7adcc1051d01e8cf not found: ID does not exist"
Oct 30 22:49:51.896792 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.896785    4947 scope.go:122] "RemoveContainer" containerID="af437a8f80bc37295ec3b64685ef7edd8918cb05c4f0f447ec032f37afa4cbfa"
Oct 30 22:49:51.897001 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:49:51.896985    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af437a8f80bc37295ec3b64685ef7edd8918cb05c4f0f447ec032f37afa4cbfa\": container with ID starting with af437a8f80bc37295ec3b64685ef7edd8918cb05c4f0f447ec032f37afa4cbfa not found: ID does not exist" containerID="af437a8f80bc37295ec3b64685ef7edd8918cb05c4f0f447ec032f37afa4cbfa"
Oct 30 22:49:51.897035 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:51.897005    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af437a8f80bc37295ec3b64685ef7edd8918cb05c4f0f447ec032f37afa4cbfa"} err="failed to get container status \"af437a8f80bc37295ec3b64685ef7edd8918cb05c4f0f447ec032f37afa4cbfa\": rpc error: code = NotFound desc = could not find container \"af437a8f80bc37295ec3b64685ef7edd8918cb05c4f0f447ec032f37afa4cbfa\": container with ID starting with af437a8f80bc37295ec3b64685ef7edd8918cb05c4f0f447ec032f37afa4cbfa not found: ID does not exist"
Oct 30 22:49:52.040172 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:52.040141    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"]
Oct 30 22:49:52.089867 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:52.089840    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-autogluon-ts-v1-noslash-predictor-67c5fd789c-45grg"]
Oct 30 22:49:52.158490 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:52.158423    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" path="/var/lib/kubelet/pods/5c0da5b5-e0f3-44ee-a47e-7b67f78b2248/volumes"
Oct 30 22:49:53.882432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:53.882399    4947 generic.go:356] "Generic (PLEG): container finished" podID="fbba0f25-356f-451a-89c2-f3a2e84c0ba2" containerID="2373c2d9f34dc41acf98dcb04a4e578cc07640458ab77f0bec88ac847c908639" exitCode=0
Oct 30 22:49:53.882807 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:53.882474    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq" event={"ID":"fbba0f25-356f-451a-89c2-f3a2e84c0ba2","Type":"ContainerDied","Data":"2373c2d9f34dc41acf98dcb04a4e578cc07640458ab77f0bec88ac847c908639"}
Oct 30 22:49:54.642569 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.642540    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"]
Oct 30 22:49:54.642877 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.642865    4947 cpu_manager.go:414] "containerMap: removing container" podUID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" containerName="kube-rbac-proxy"
Oct 30 22:49:54.642934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.642878    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" containerName="kube-rbac-proxy"
Oct 30 22:49:54.642934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.642889    4947 cpu_manager.go:414] "containerMap: removing container" podUID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" containerName="kserve-container"
Oct 30 22:49:54.642934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.642894    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" containerName="kserve-container"
Oct 30 22:49:54.642934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.642910    4947 cpu_manager.go:414] "containerMap: removing container" podUID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" containerName="storage-initializer"
Oct 30 22:49:54.642934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.642915    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" containerName="storage-initializer"
Oct 30 22:49:54.643099 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.642967    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" containerName="kserve-container"
Oct 30 22:49:54.643099 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.642980    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c0da5b5-e0f3-44ee-a47e-7b67f78b2248" containerName="kube-rbac-proxy"
Oct 30 22:49:54.645157 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.645139    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:49:54.698409 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.698379    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh2w8\" (UniqueName: \"kubernetes.io/projected/f0f4236b-b86e-467a-b0d4-01139e70d777-kube-api-access-wh2w8\") pod \"isvc-canary-create-v2-predictor-76d6db7cf9-r28xv\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:49:54.698552 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.698418    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0f4236b-b86e-467a-b0d4-01139e70d777-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-76d6db7cf9-r28xv\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:49:54.698552 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.698525    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0f4236b-b86e-467a-b0d4-01139e70d777-proxy-tls\") pod \"isvc-canary-create-v2-predictor-76d6db7cf9-r28xv\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:49:54.698624 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.698550    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0f4236b-b86e-467a-b0d4-01139e70d777-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-76d6db7cf9-r28xv\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:49:54.698624 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.698582    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0f4236b-b86e-467a-b0d4-01139e70d777-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-76d6db7cf9-r28xv\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:49:54.699764 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.699745    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"seaweedfs-s3-creds\""
Oct 30 22:49:54.700291 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.700276    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Oct 30 22:49:54.700349 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.700315    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"default-dockercfg-s9cqw\""
Oct 30 22:49:54.701838 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.701825    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:49:54.701874 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.701847    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Oct 30 22:49:54.701919 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.701908    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"openshift-service-ca.crt\""
Oct 30 22:49:54.702076 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.702065    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"kube-root-ca.crt\""
Oct 30 22:49:54.741908 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.741885    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"]
Oct 30 22:49:54.799307 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.799277    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh2w8\" (UniqueName: \"kubernetes.io/projected/f0f4236b-b86e-467a-b0d4-01139e70d777-kube-api-access-wh2w8\") pod \"isvc-canary-create-v2-predictor-76d6db7cf9-r28xv\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:49:54.799473 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.799311    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0f4236b-b86e-467a-b0d4-01139e70d777-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-76d6db7cf9-r28xv\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:49:54.799473 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.799370    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0f4236b-b86e-467a-b0d4-01139e70d777-proxy-tls\") pod \"isvc-canary-create-v2-predictor-76d6db7cf9-r28xv\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:49:54.799473 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.799389    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0f4236b-b86e-467a-b0d4-01139e70d777-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-76d6db7cf9-r28xv\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:49:54.799623 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.799502    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0f4236b-b86e-467a-b0d4-01139e70d777-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-76d6db7cf9-r28xv\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:49:54.799887 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.799862    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0f4236b-b86e-467a-b0d4-01139e70d777-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-76d6db7cf9-r28xv\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:49:54.800198 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.800174    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0f4236b-b86e-467a-b0d4-01139e70d777-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-76d6db7cf9-r28xv\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:49:54.800198 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.800193    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0f4236b-b86e-467a-b0d4-01139e70d777-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-76d6db7cf9-r28xv\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:49:54.802053 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.802031    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0f4236b-b86e-467a-b0d4-01139e70d777-proxy-tls\") pod \"isvc-canary-create-v2-predictor-76d6db7cf9-r28xv\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:49:54.862344 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.862297    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh2w8\" (UniqueName: \"kubernetes.io/projected/f0f4236b-b86e-467a-b0d4-01139e70d777-kube-api-access-wh2w8\") pod \"isvc-canary-create-v2-predictor-76d6db7cf9-r28xv\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:49:54.888673 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.888641    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq" event={"ID":"fbba0f25-356f-451a-89c2-f3a2e84c0ba2","Type":"ContainerStarted","Data":"0cd34d1a80b7272266448e73776e499a9d87b98317661b3212d8abe151ffdac6"}
Oct 30 22:49:54.888991 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.888701    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq" event={"ID":"fbba0f25-356f-451a-89c2-f3a2e84c0ba2","Type":"ContainerStarted","Data":"0783267f2af615dda30a48f3ba110e5c2a05e6de4988d3b3d75facb518f473fd"}
Oct 30 22:49:54.888991 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.888888    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:54.959237 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:54.959169    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:49:55.195191 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.195144    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq" podStartSLOduration=3.244543136 podStartE2EDuration="7.19512589s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.950582754s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:55.011787635 +0000 UTC m=+2493.011135273" watchObservedRunningTime="2026-10-30 22:49:55.19512589 +0000 UTC m=+2493.194473454"
Oct 30 22:49:55.195997 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.195979    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"]
Oct 30 22:49:55.198750 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.198734    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:49:55.203355 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.203299    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhqbw\" (UniqueName: \"kubernetes.io/projected/c3045563-20c5-4655-b1c1-2d225a590586-kube-api-access-qhqbw\") pod \"isvc-canary-create-predictor-7d65b89964-d7xbs\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:49:55.203492 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.203386    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3045563-20c5-4655-b1c1-2d225a590586-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d65b89964-d7xbs\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:49:55.203492 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.203442    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3045563-20c5-4655-b1c1-2d225a590586-proxy-tls\") pod \"isvc-canary-create-predictor-7d65b89964-d7xbs\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:49:55.203625 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.203507    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3045563-20c5-4655-b1c1-2d225a590586-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d65b89964-d7xbs\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:49:55.203625 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.203549    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3045563-20c5-4655-b1c1-2d225a590586-cabundle-cert\") pod \"isvc-canary-create-predictor-7d65b89964-d7xbs\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:49:55.252561 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.252510    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-canary-create-predictor-serving-cert\""
Oct 30 22:49:55.262443 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.262417    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"]
Oct 30 22:49:55.304619 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.304591    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhqbw\" (UniqueName: \"kubernetes.io/projected/c3045563-20c5-4655-b1c1-2d225a590586-kube-api-access-qhqbw\") pod \"isvc-canary-create-predictor-7d65b89964-d7xbs\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:49:55.304749 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.304622    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3045563-20c5-4655-b1c1-2d225a590586-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d65b89964-d7xbs\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:49:55.304749 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.304649    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3045563-20c5-4655-b1c1-2d225a590586-proxy-tls\") pod \"isvc-canary-create-predictor-7d65b89964-d7xbs\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:49:55.304749 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.304684    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3045563-20c5-4655-b1c1-2d225a590586-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d65b89964-d7xbs\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:49:55.304749 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.304705    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3045563-20c5-4655-b1c1-2d225a590586-cabundle-cert\") pod \"isvc-canary-create-predictor-7d65b89964-d7xbs\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:49:55.305077 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.305057    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3045563-20c5-4655-b1c1-2d225a590586-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d65b89964-d7xbs\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:49:55.305419 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.305403    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3045563-20c5-4655-b1c1-2d225a590586-cabundle-cert\") pod \"isvc-canary-create-predictor-7d65b89964-d7xbs\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:49:55.305509 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.305492    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3045563-20c5-4655-b1c1-2d225a590586-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d65b89964-d7xbs\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:49:55.307298 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.307282    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3045563-20c5-4655-b1c1-2d225a590586-proxy-tls\") pod \"isvc-canary-create-predictor-7d65b89964-d7xbs\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:49:55.378835 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.378811    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhqbw\" (UniqueName: \"kubernetes.io/projected/c3045563-20c5-4655-b1c1-2d225a590586-kube-api-access-qhqbw\") pod \"isvc-canary-create-predictor-7d65b89964-d7xbs\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") " pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:49:55.390799 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.390672    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"]
Oct 30 22:49:55.393258 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:49:55.393233    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f4236b_b86e_467a_b0d4_01139e70d777.slice/crio-2132a71ee030ea0182c472d03db3595999c879b0f407961bd55000215400db58 WatchSource:0}: Error finding container 2132a71ee030ea0182c472d03db3595999c879b0f407961bd55000215400db58: Status 404 returned error can't find the container with id 2132a71ee030ea0182c472d03db3595999c879b0f407961bd55000215400db58
Oct 30 22:49:55.512427 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.512326    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:49:55.885465 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.885431    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"]
Oct 30 22:49:55.888865 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:49:55.888836    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3045563_20c5_4655_b1c1_2d225a590586.slice/crio-3d172a0b0ca4975ddd1cf24a8cfe3a821dca90973425710d9b419b55caeb8c38 WatchSource:0}: Error finding container 3d172a0b0ca4975ddd1cf24a8cfe3a821dca90973425710d9b419b55caeb8c38: Status 404 returned error can't find the container with id 3d172a0b0ca4975ddd1cf24a8cfe3a821dca90973425710d9b419b55caeb8c38
Oct 30 22:49:55.895326 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.895268    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" event={"ID":"f0f4236b-b86e-467a-b0d4-01139e70d777","Type":"ContainerStarted","Data":"96ac1e99480bc6d8a56e9fecd5122a8f944e69b27a5494d3bf4d5add68305e0c"}
Oct 30 22:49:55.895326 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.895368    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:49:55.895586 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:55.895408    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" event={"ID":"f0f4236b-b86e-467a-b0d4-01139e70d777","Type":"ContainerStarted","Data":"2132a71ee030ea0182c472d03db3595999c879b0f407961bd55000215400db58"}
Oct 30 22:49:56.901112 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:56.901079    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" event={"ID":"c3045563-20c5-4655-b1c1-2d225a590586","Type":"ContainerStarted","Data":"5f77232bd34628f0f4bb2344f4a3bb41944c6b722318fc357ad5d82966ba2ba2"}
Oct 30 22:49:56.901623 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:56.901135    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" event={"ID":"c3045563-20c5-4655-b1c1-2d225a590586","Type":"ContainerStarted","Data":"3d172a0b0ca4975ddd1cf24a8cfe3a821dca90973425710d9b419b55caeb8c38"}
Oct 30 22:49:59.048891 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:49:59.048859    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:49:59.914770 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:59.914734    4947 generic.go:356] "Generic (PLEG): container finished" podID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerID="96ac1e99480bc6d8a56e9fecd5122a8f944e69b27a5494d3bf4d5add68305e0c" exitCode=0
Oct 30 22:49:59.914917 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:49:59.914809    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" event={"ID":"f0f4236b-b86e-467a-b0d4-01139e70d777","Type":"ContainerDied","Data":"96ac1e99480bc6d8a56e9fecd5122a8f944e69b27a5494d3bf4d5add68305e0c"}
Oct 30 22:50:00.921816 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:00.921780    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" event={"ID":"f0f4236b-b86e-467a-b0d4-01139e70d777","Type":"ContainerStarted","Data":"0c7d986e517176cd747b8258152b6753daa860588b17966b2a6f13fc5bad3cd0"}
Oct 30 22:50:00.922281 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:00.921850    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" event={"ID":"f0f4236b-b86e-467a-b0d4-01139e70d777","Type":"ContainerStarted","Data":"23033427d653ecdeef1748a4ca36c17c71fe2c2b07cc39ab2d01470f6122dbc1"}
Oct 30 22:50:00.922281 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:00.922180    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:50:00.922433 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:00.922416    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:50:00.923447 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:00.923417    4947 generic.go:356] "Generic (PLEG): container finished" podID="c3045563-20c5-4655-b1c1-2d225a590586" containerID="5f77232bd34628f0f4bb2344f4a3bb41944c6b722318fc357ad5d82966ba2ba2" exitCode=0
Oct 30 22:50:00.923574 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:00.923455    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" event={"ID":"c3045563-20c5-4655-b1c1-2d225a590586","Type":"ContainerDied","Data":"5f77232bd34628f0f4bb2344f4a3bb41944c6b722318fc357ad5d82966ba2ba2"}
Oct 30 22:50:00.923865 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:00.923841    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Oct 30 22:50:01.048029 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:01.047984    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" podStartSLOduration=3.212509738 podStartE2EDuration="7.047968981s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.835459243s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:01.047267109 +0000 UTC m=+2499.046614671" watchObservedRunningTime="2026-10-30 22:50:01.047968981 +0000 UTC m=+2499.047316544"
Oct 30 22:50:01.908060 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:01.908031    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:50:01.930180 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:01.930145    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" event={"ID":"c3045563-20c5-4655-b1c1-2d225a590586","Type":"ContainerStarted","Data":"a50eb2448fbbf2e0c1855c411a54114e7298dd8b3b6fb459a8786e358150f9b2"}
Oct 30 22:50:01.930598 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:01.930208    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" event={"ID":"c3045563-20c5-4655-b1c1-2d225a590586","Type":"ContainerStarted","Data":"59300f28a99da412dda92b2752e3d514db7745b1e3feb60cd3089f4c6e76f5a6"}
Oct 30 22:50:01.930598 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:01.930518    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Oct 30 22:50:01.930598 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:01.930587    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:50:01.930790 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:01.930632    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:50:01.931717 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:01.931694    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 30 22:50:02.153692 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:02.153634    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" podStartSLOduration=3.182015306 podStartE2EDuration="7.153617436s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.97160213s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:02.151084041 +0000 UTC m=+2500.150431605" watchObservedRunningTime="2026-10-30 22:50:02.153617436 +0000 UTC m=+2500.152965000"
Oct 30 22:50:02.934724 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:02.934686    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 30 22:50:06.935317 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:06.935285    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:50:06.935850 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:06.935825    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Oct 30 22:50:07.939955 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:07.939923    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:50:07.940628 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:07.940599    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 30 22:50:16.936034 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:16.935989    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Oct 30 22:50:17.940643 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:17.940599    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 30 22:50:26.936009 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:26.935970    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Oct 30 22:50:27.941391 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:27.941354    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 30 22:50:29.047758 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:50:29.047722    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:50:31.998040 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:31.997994    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:50:36.935970 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:36.935931    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Oct 30 22:50:37.940585 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:37.940543    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 30 22:50:38.703746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:38.703707    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"]
Oct 30 22:50:38.703979 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:38.703955    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq" podUID="fbba0f25-356f-451a-89c2-f3a2e84c0ba2" containerName="kserve-container" containerID="cri-o://0783267f2af615dda30a48f3ba110e5c2a05e6de4988d3b3d75facb518f473fd" gracePeriod=30
Oct 30 22:50:38.704138 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:38.704066    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq" podUID="fbba0f25-356f-451a-89c2-f3a2e84c0ba2" containerName="kube-rbac-proxy" containerID="cri-o://0cd34d1a80b7272266448e73776e499a9d87b98317661b3212d8abe151ffdac6" gracePeriod=30
Oct 30 22:50:39.090733 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:39.090700    4947 generic.go:356] "Generic (PLEG): container finished" podID="fbba0f25-356f-451a-89c2-f3a2e84c0ba2" containerID="0cd34d1a80b7272266448e73776e499a9d87b98317661b3212d8abe151ffdac6" exitCode=0
Oct 30 22:50:39.091107 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:39.090752    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq" event={"ID":"fbba0f25-356f-451a-89c2-f3a2e84c0ba2","Type":"ContainerDied","Data":"0cd34d1a80b7272266448e73776e499a9d87b98317661b3212d8abe151ffdac6"}
Oct 30 22:50:41.901754 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:41.901712    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq" podUID="fbba0f25-356f-451a-89c2-f3a2e84c0ba2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Oct 30 22:50:41.909296 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:41.909258    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq" podUID="fbba0f25-356f-451a-89c2-f3a2e84c0ba2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.57:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.132.0.57:8080: connect: connection refused"
Oct 30 22:50:45.747205 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:45.747172    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:50:45.921291 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:45.921033    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kube-api-access-btdw4\" (UniqueName: \"kubernetes.io/projected/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kube-api-access-btdw4\") pod \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") "
Oct 30 22:50:45.921291 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:45.921098    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kserve-provision-location\") pod \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") "
Oct 30 22:50:45.921291 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:45.921212    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-cabundle-cert\") pod \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") "
Oct 30 22:50:45.921291 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:45.921305    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kube-rbac-proxy-sar-config\") pod \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") "
Oct 30 22:50:45.921742 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:45.921360    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-proxy-tls\") pod \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\" (UID: \"fbba0f25-356f-451a-89c2-f3a2e84c0ba2\") "
Oct 30 22:50:45.921939 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:45.921894    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kube-rbac-proxy-sar-config" pod "fbba0f25-356f-451a-89c2-f3a2e84c0ba2" (UID: "fbba0f25-356f-451a-89c2-f3a2e84c0ba2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:45.922076 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:45.921950    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-cabundle-cert" pod "fbba0f25-356f-451a-89c2-f3a2e84c0ba2" (UID: "fbba0f25-356f-451a-89c2-f3a2e84c0ba2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:45.922193 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:45.922156    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kserve-provision-location" pod "fbba0f25-356f-451a-89c2-f3a2e84c0ba2" (UID: "fbba0f25-356f-451a-89c2-f3a2e84c0ba2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:45.924168 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:45.924142    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-proxy-tls" pod "fbba0f25-356f-451a-89c2-f3a2e84c0ba2" (UID: "fbba0f25-356f-451a-89c2-f3a2e84c0ba2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:45.924293 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:45.924271    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kube-api-access-btdw4" pod "fbba0f25-356f-451a-89c2-f3a2e84c0ba2" (UID: "fbba0f25-356f-451a-89c2-f3a2e84c0ba2"). InnerVolumeSpecName "kube-api-access-btdw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:46.022205 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.022166    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.022205 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.022198    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.022205 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.022207    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.022472 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.022217    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.022472 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.022228    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btdw4\" (UniqueName: \"kubernetes.io/projected/fbba0f25-356f-451a-89c2-f3a2e84c0ba2-kube-api-access-btdw4\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.121578 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.121539    4947 generic.go:356] "Generic (PLEG): container finished" podID="fbba0f25-356f-451a-89c2-f3a2e84c0ba2" containerID="0783267f2af615dda30a48f3ba110e5c2a05e6de4988d3b3d75facb518f473fd" exitCode=0
Oct 30 22:50:46.121741 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.121617    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"
Oct 30 22:50:46.121741 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.121614    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq" event={"ID":"fbba0f25-356f-451a-89c2-f3a2e84c0ba2","Type":"ContainerDied","Data":"0783267f2af615dda30a48f3ba110e5c2a05e6de4988d3b3d75facb518f473fd"}
Oct 30 22:50:46.121817 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.121738    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq" event={"ID":"fbba0f25-356f-451a-89c2-f3a2e84c0ba2","Type":"ContainerDied","Data":"d5a7068d1111b4d5d2e85727469a7f5ab9ad2ee35222c4603c5fd7a80ea8197e"}
Oct 30 22:50:46.121817 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.121770    4947 scope.go:122] "RemoveContainer" containerID="0cd34d1a80b7272266448e73776e499a9d87b98317661b3212d8abe151ffdac6"
Oct 30 22:50:46.130489 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.130469    4947 scope.go:122] "RemoveContainer" containerID="0783267f2af615dda30a48f3ba110e5c2a05e6de4988d3b3d75facb518f473fd"
Oct 30 22:50:46.137969 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.137949    4947 scope.go:122] "RemoveContainer" containerID="2373c2d9f34dc41acf98dcb04a4e578cc07640458ab77f0bec88ac847c908639"
Oct 30 22:50:46.145475 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.145457    4947 scope.go:122] "RemoveContainer" containerID="0cd34d1a80b7272266448e73776e499a9d87b98317661b3212d8abe151ffdac6"
Oct 30 22:50:46.145735 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:50:46.145713    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cd34d1a80b7272266448e73776e499a9d87b98317661b3212d8abe151ffdac6\": container with ID starting with 0cd34d1a80b7272266448e73776e499a9d87b98317661b3212d8abe151ffdac6 not found: ID does not exist" containerID="0cd34d1a80b7272266448e73776e499a9d87b98317661b3212d8abe151ffdac6"
Oct 30 22:50:46.145794 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.145749    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd34d1a80b7272266448e73776e499a9d87b98317661b3212d8abe151ffdac6"} err="failed to get container status \"0cd34d1a80b7272266448e73776e499a9d87b98317661b3212d8abe151ffdac6\": rpc error: code = NotFound desc = could not find container \"0cd34d1a80b7272266448e73776e499a9d87b98317661b3212d8abe151ffdac6\": container with ID starting with 0cd34d1a80b7272266448e73776e499a9d87b98317661b3212d8abe151ffdac6 not found: ID does not exist"
Oct 30 22:50:46.145794 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.145776    4947 scope.go:122] "RemoveContainer" containerID="0783267f2af615dda30a48f3ba110e5c2a05e6de4988d3b3d75facb518f473fd"
Oct 30 22:50:46.145989 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:50:46.145974    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0783267f2af615dda30a48f3ba110e5c2a05e6de4988d3b3d75facb518f473fd\": container with ID starting with 0783267f2af615dda30a48f3ba110e5c2a05e6de4988d3b3d75facb518f473fd not found: ID does not exist" containerID="0783267f2af615dda30a48f3ba110e5c2a05e6de4988d3b3d75facb518f473fd"
Oct 30 22:50:46.146047 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.145994    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0783267f2af615dda30a48f3ba110e5c2a05e6de4988d3b3d75facb518f473fd"} err="failed to get container status \"0783267f2af615dda30a48f3ba110e5c2a05e6de4988d3b3d75facb518f473fd\": rpc error: code = NotFound desc = could not find container \"0783267f2af615dda30a48f3ba110e5c2a05e6de4988d3b3d75facb518f473fd\": container with ID starting with 0783267f2af615dda30a48f3ba110e5c2a05e6de4988d3b3d75facb518f473fd not found: ID does not exist"
Oct 30 22:50:46.146047 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.146006    4947 scope.go:122] "RemoveContainer" containerID="2373c2d9f34dc41acf98dcb04a4e578cc07640458ab77f0bec88ac847c908639"
Oct 30 22:50:46.146234 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:50:46.146214    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2373c2d9f34dc41acf98dcb04a4e578cc07640458ab77f0bec88ac847c908639\": container with ID starting with 2373c2d9f34dc41acf98dcb04a4e578cc07640458ab77f0bec88ac847c908639 not found: ID does not exist" containerID="2373c2d9f34dc41acf98dcb04a4e578cc07640458ab77f0bec88ac847c908639"
Oct 30 22:50:46.146291 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.146244    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2373c2d9f34dc41acf98dcb04a4e578cc07640458ab77f0bec88ac847c908639"} err="failed to get container status \"2373c2d9f34dc41acf98dcb04a4e578cc07640458ab77f0bec88ac847c908639\": rpc error: code = NotFound desc = could not find container \"2373c2d9f34dc41acf98dcb04a4e578cc07640458ab77f0bec88ac847c908639\": container with ID starting with 2373c2d9f34dc41acf98dcb04a4e578cc07640458ab77f0bec88ac847c908639 not found: ID does not exist"
Oct 30 22:50:46.289989 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.289958    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"]
Oct 30 22:50:46.295489 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.295468    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-mlserver-predictor-75f95498df-mljvq"]
Oct 30 22:50:46.936305 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:46.936262    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Oct 30 22:50:47.941240 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:47.941202    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 30 22:50:48.157553 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:48.157516    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbba0f25-356f-451a-89c2-f3a2e84c0ba2" path="/var/lib/kubelet/pods/fbba0f25-356f-451a-89c2-f3a2e84c0ba2/volumes"
Oct 30 22:50:49.804899 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.804854    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"]
Oct 30 22:50:49.805410 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.805224    4947 cpu_manager.go:414] "containerMap: removing container" podUID="fbba0f25-356f-451a-89c2-f3a2e84c0ba2" containerName="kube-rbac-proxy"
Oct 30 22:50:49.805410 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.805237    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbba0f25-356f-451a-89c2-f3a2e84c0ba2" containerName="kube-rbac-proxy"
Oct 30 22:50:49.805410 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.805251    4947 cpu_manager.go:414] "containerMap: removing container" podUID="fbba0f25-356f-451a-89c2-f3a2e84c0ba2" containerName="kserve-container"
Oct 30 22:50:49.805410 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.805259    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbba0f25-356f-451a-89c2-f3a2e84c0ba2" containerName="kserve-container"
Oct 30 22:50:49.805410 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.805274    4947 cpu_manager.go:414] "containerMap: removing container" podUID="fbba0f25-356f-451a-89c2-f3a2e84c0ba2" containerName="storage-initializer"
Oct 30 22:50:49.805410 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.805281    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbba0f25-356f-451a-89c2-f3a2e84c0ba2" containerName="storage-initializer"
Oct 30 22:50:49.805410 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.805364    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbba0f25-356f-451a-89c2-f3a2e84c0ba2" containerName="kube-rbac-proxy"
Oct 30 22:50:49.805410 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.805373    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbba0f25-356f-451a-89c2-f3a2e84c0ba2" containerName="kserve-container"
Oct 30 22:50:49.808642 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.808626    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:49.848364 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.848319    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5zhw\" (UniqueName: \"kubernetes.io/projected/fe522781-a66c-4290-a371-9340c1be0883-kube-api-access-j5zhw\") pod \"xgboost-v2-mlserver-predictor-84697fb8cb-tbn84\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") " pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:49.848483 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.848371    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe522781-a66c-4290-a371-9340c1be0883-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-84697fb8cb-tbn84\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") " pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:49.848483 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.848447    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe522781-a66c-4290-a371-9340c1be0883-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-84697fb8cb-tbn84\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") " pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:49.848571 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.848516    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe522781-a66c-4290-a371-9340c1be0883-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-84697fb8cb-tbn84\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") " pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:49.848571 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.848551    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe522781-a66c-4290-a371-9340c1be0883-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-84697fb8cb-tbn84\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") " pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:49.863596 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.863575    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:50:49.912513 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.912481    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:50:49.912666 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.912593    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Oct 30 22:50:49.912729 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.912668    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 30 22:50:49.912729 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.912688    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:50:49.912729 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.912706    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:50:49.912729 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.912720    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:50:49.914203 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.914183    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"]
Oct 30 22:50:49.949032 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.949001    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe522781-a66c-4290-a371-9340c1be0883-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-84697fb8cb-tbn84\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") " pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:49.949171 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.949057    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe522781-a66c-4290-a371-9340c1be0883-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-84697fb8cb-tbn84\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") " pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:49.949171 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.949100    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5zhw\" (UniqueName: \"kubernetes.io/projected/fe522781-a66c-4290-a371-9340c1be0883-kube-api-access-j5zhw\") pod \"xgboost-v2-mlserver-predictor-84697fb8cb-tbn84\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") " pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:49.949171 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.949130    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe522781-a66c-4290-a371-9340c1be0883-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-84697fb8cb-tbn84\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") " pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:49.949171 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.949165    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe522781-a66c-4290-a371-9340c1be0883-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-84697fb8cb-tbn84\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") " pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:49.949588 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.949569    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe522781-a66c-4290-a371-9340c1be0883-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-84697fb8cb-tbn84\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") " pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:49.949882 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.949861    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe522781-a66c-4290-a371-9340c1be0883-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-84697fb8cb-tbn84\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") " pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:49.949925 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.949861    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe522781-a66c-4290-a371-9340c1be0883-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-84697fb8cb-tbn84\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") " pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:49.951751 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:49.951730    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe522781-a66c-4290-a371-9340c1be0883-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-84697fb8cb-tbn84\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") " pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:50.012364 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:50.012317    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5zhw\" (UniqueName: \"kubernetes.io/projected/fe522781-a66c-4290-a371-9340c1be0883-kube-api-access-j5zhw\") pod \"xgboost-v2-mlserver-predictor-84697fb8cb-tbn84\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") " pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:50.125911 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:50.125816    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:50.554603 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:50.554571    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"]
Oct 30 22:50:50.557710 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:50:50.557679    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe522781_a66c_4290_a371_9340c1be0883.slice/crio-8ab66aa17ac4db2d67bdd5cff53c1ac9d1c3223f6303e703f5fdccde9b97a241 WatchSource:0}: Error finding container 8ab66aa17ac4db2d67bdd5cff53c1ac9d1c3223f6303e703f5fdccde9b97a241: Status 404 returned error can't find the container with id 8ab66aa17ac4db2d67bdd5cff53c1ac9d1c3223f6303e703f5fdccde9b97a241
Oct 30 22:50:51.146212 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:51.146177    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84" event={"ID":"fe522781-a66c-4290-a371-9340c1be0883","Type":"ContainerStarted","Data":"30d4bab49f9d3357114e4dfb8565ff3a6705a9d46973c3cb53feea259272b197"}
Oct 30 22:50:51.146647 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:51.146234    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84" event={"ID":"fe522781-a66c-4290-a371-9340c1be0883","Type":"ContainerStarted","Data":"8ab66aa17ac4db2d67bdd5cff53c1ac9d1c3223f6303e703f5fdccde9b97a241"}
Oct 30 22:50:55.163536 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:55.163504    4947 generic.go:356] "Generic (PLEG): container finished" podID="fe522781-a66c-4290-a371-9340c1be0883" containerID="30d4bab49f9d3357114e4dfb8565ff3a6705a9d46973c3cb53feea259272b197" exitCode=0
Oct 30 22:50:55.163926 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:55.163579    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84" event={"ID":"fe522781-a66c-4290-a371-9340c1be0883","Type":"ContainerDied","Data":"30d4bab49f9d3357114e4dfb8565ff3a6705a9d46973c3cb53feea259272b197"}
Oct 30 22:50:56.170080 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:56.170042    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84" event={"ID":"fe522781-a66c-4290-a371-9340c1be0883","Type":"ContainerStarted","Data":"c143c8e37e9ea1f67f1d77c9a6b8eea6381481fd233b89770ad3ba18938bc045"}
Oct 30 22:50:56.170528 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:56.170100    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84" event={"ID":"fe522781-a66c-4290-a371-9340c1be0883","Type":"ContainerStarted","Data":"31aa28c9ee572daaec29fc894b4af77c576a525857bf530b943f5dae983e25b4"}
Oct 30 22:50:56.170528 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:56.170405    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:56.170528 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:56.170459    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:50:56.296305 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:56.296241    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84" podStartSLOduration=3.195972103 podStartE2EDuration="7.29622235s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.100250247s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:56.295056464 +0000 UTC m=+2554.294404028" watchObservedRunningTime="2026-10-30 22:50:56.29622235 +0000 UTC m=+2554.295569914"
Oct 30 22:50:56.936062 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:56.936023    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Oct 30 22:50:57.941275 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:50:57.941235    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 30 22:50:59.048598 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:50:59.048557    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:51:02.180435 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:02.180355    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:51:06.936299 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:06.936259    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Oct 30 22:51:07.940498 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:07.940457    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 30 22:51:16.936579 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:16.936544    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:51:17.940863 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:17.940818    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 30 22:51:18.384637 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:18.384602    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"]
Oct 30 22:51:18.384893 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:18.384846    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kserve-container" containerID="cri-o://59300f28a99da412dda92b2752e3d514db7745b1e3feb60cd3089f4c6e76f5a6" gracePeriod=30
Oct 30 22:51:18.385048 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:18.384947    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kube-rbac-proxy" containerID="cri-o://a50eb2448fbbf2e0c1855c411a54114e7298dd8b3b6fb459a8786e358150f9b2" gracePeriod=30
Oct 30 22:51:18.385110 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:18.385059    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 30 22:51:19.270422 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:19.270315    4947 generic.go:356] "Generic (PLEG): container finished" podID="c3045563-20c5-4655-b1c1-2d225a590586" containerID="a50eb2448fbbf2e0c1855c411a54114e7298dd8b3b6fb459a8786e358150f9b2" exitCode=0
Oct 30 22:51:19.270422 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:19.270387    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" event={"ID":"c3045563-20c5-4655-b1c1-2d225a590586","Type":"ContainerDied","Data":"a50eb2448fbbf2e0c1855c411a54114e7298dd8b3b6fb459a8786e358150f9b2"}
Oct 30 22:51:22.935261 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:22.935221    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Oct 30 22:51:23.631492 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:23.631469    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:51:23.738707 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:23.738676    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3045563-20c5-4655-b1c1-2d225a590586-kube-api-access-qhqbw\" (UniqueName: \"kubernetes.io/projected/c3045563-20c5-4655-b1c1-2d225a590586-kube-api-access-qhqbw\") pod \"c3045563-20c5-4655-b1c1-2d225a590586\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") "
Oct 30 22:51:23.738707 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:23.738707    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3045563-20c5-4655-b1c1-2d225a590586-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3045563-20c5-4655-b1c1-2d225a590586-cabundle-cert\") pod \"c3045563-20c5-4655-b1c1-2d225a590586\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") "
Oct 30 22:51:23.738891 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:23.738834    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3045563-20c5-4655-b1c1-2d225a590586-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3045563-20c5-4655-b1c1-2d225a590586-kserve-provision-location\") pod \"c3045563-20c5-4655-b1c1-2d225a590586\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") "
Oct 30 22:51:23.738930 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:23.738902    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3045563-20c5-4655-b1c1-2d225a590586-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3045563-20c5-4655-b1c1-2d225a590586-proxy-tls\") pod \"c3045563-20c5-4655-b1c1-2d225a590586\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") "
Oct 30 22:51:23.738977 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:23.738932    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3045563-20c5-4655-b1c1-2d225a590586-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3045563-20c5-4655-b1c1-2d225a590586-kube-rbac-proxy-sar-config\") pod \"c3045563-20c5-4655-b1c1-2d225a590586\" (UID: \"c3045563-20c5-4655-b1c1-2d225a590586\") "
Oct 30 22:51:23.739131 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:23.739100    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3045563-20c5-4655-b1c1-2d225a590586-cabundle-cert" pod "c3045563-20c5-4655-b1c1-2d225a590586" (UID: "c3045563-20c5-4655-b1c1-2d225a590586"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:23.739131 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:23.739109    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3045563-20c5-4655-b1c1-2d225a590586-kserve-provision-location" pod "c3045563-20c5-4655-b1c1-2d225a590586" (UID: "c3045563-20c5-4655-b1c1-2d225a590586"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:23.739297 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:23.739190    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3045563-20c5-4655-b1c1-2d225a590586-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:23.739297 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:23.739207    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3045563-20c5-4655-b1c1-2d225a590586-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:23.739406 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:23.739323    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3045563-20c5-4655-b1c1-2d225a590586-kube-rbac-proxy-sar-config" pod "c3045563-20c5-4655-b1c1-2d225a590586" (UID: "c3045563-20c5-4655-b1c1-2d225a590586"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:23.740976 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:23.740949    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3045563-20c5-4655-b1c1-2d225a590586-kube-api-access-qhqbw" pod "c3045563-20c5-4655-b1c1-2d225a590586" (UID: "c3045563-20c5-4655-b1c1-2d225a590586"). InnerVolumeSpecName "kube-api-access-qhqbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:23.741059 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:23.741005    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3045563-20c5-4655-b1c1-2d225a590586-proxy-tls" pod "c3045563-20c5-4655-b1c1-2d225a590586" (UID: "c3045563-20c5-4655-b1c1-2d225a590586"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:23.840304 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:23.840229    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhqbw\" (UniqueName: \"kubernetes.io/projected/c3045563-20c5-4655-b1c1-2d225a590586-kube-api-access-qhqbw\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:23.840304 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:23.840256    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3045563-20c5-4655-b1c1-2d225a590586-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:23.840304 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:23.840266    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3045563-20c5-4655-b1c1-2d225a590586-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:24.292242 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:24.292205    4947 generic.go:356] "Generic (PLEG): container finished" podID="c3045563-20c5-4655-b1c1-2d225a590586" containerID="59300f28a99da412dda92b2752e3d514db7745b1e3feb60cd3089f4c6e76f5a6" exitCode=139
Oct 30 22:51:24.292742 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:24.292273    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"
Oct 30 22:51:24.292742 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:24.292294    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" event={"ID":"c3045563-20c5-4655-b1c1-2d225a590586","Type":"ContainerDied","Data":"59300f28a99da412dda92b2752e3d514db7745b1e3feb60cd3089f4c6e76f5a6"}
Oct 30 22:51:24.292742 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:24.292396    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs" event={"ID":"c3045563-20c5-4655-b1c1-2d225a590586","Type":"ContainerDied","Data":"3d172a0b0ca4975ddd1cf24a8cfe3a821dca90973425710d9b419b55caeb8c38"}
Oct 30 22:51:24.292742 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:24.292451    4947 scope.go:122] "RemoveContainer" containerID="a50eb2448fbbf2e0c1855c411a54114e7298dd8b3b6fb459a8786e358150f9b2"
Oct 30 22:51:24.300959 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:24.300941    4947 scope.go:122] "RemoveContainer" containerID="59300f28a99da412dda92b2752e3d514db7745b1e3feb60cd3089f4c6e76f5a6"
Oct 30 22:51:24.308269 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:24.308240    4947 scope.go:122] "RemoveContainer" containerID="5f77232bd34628f0f4bb2344f4a3bb41944c6b722318fc357ad5d82966ba2ba2"
Oct 30 22:51:24.315575 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:24.315556    4947 scope.go:122] "RemoveContainer" containerID="a50eb2448fbbf2e0c1855c411a54114e7298dd8b3b6fb459a8786e358150f9b2"
Oct 30 22:51:24.315850 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:51:24.315831    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50eb2448fbbf2e0c1855c411a54114e7298dd8b3b6fb459a8786e358150f9b2\": container with ID starting with a50eb2448fbbf2e0c1855c411a54114e7298dd8b3b6fb459a8786e358150f9b2 not found: ID does not exist" containerID="a50eb2448fbbf2e0c1855c411a54114e7298dd8b3b6fb459a8786e358150f9b2"
Oct 30 22:51:24.315928 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:24.315862    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50eb2448fbbf2e0c1855c411a54114e7298dd8b3b6fb459a8786e358150f9b2"} err="failed to get container status \"a50eb2448fbbf2e0c1855c411a54114e7298dd8b3b6fb459a8786e358150f9b2\": rpc error: code = NotFound desc = could not find container \"a50eb2448fbbf2e0c1855c411a54114e7298dd8b3b6fb459a8786e358150f9b2\": container with ID starting with a50eb2448fbbf2e0c1855c411a54114e7298dd8b3b6fb459a8786e358150f9b2 not found: ID does not exist"
Oct 30 22:51:24.315928 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:24.315890    4947 scope.go:122] "RemoveContainer" containerID="59300f28a99da412dda92b2752e3d514db7745b1e3feb60cd3089f4c6e76f5a6"
Oct 30 22:51:24.316140 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:51:24.316123    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59300f28a99da412dda92b2752e3d514db7745b1e3feb60cd3089f4c6e76f5a6\": container with ID starting with 59300f28a99da412dda92b2752e3d514db7745b1e3feb60cd3089f4c6e76f5a6 not found: ID does not exist" containerID="59300f28a99da412dda92b2752e3d514db7745b1e3feb60cd3089f4c6e76f5a6"
Oct 30 22:51:24.316177 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:24.316146    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59300f28a99da412dda92b2752e3d514db7745b1e3feb60cd3089f4c6e76f5a6"} err="failed to get container status \"59300f28a99da412dda92b2752e3d514db7745b1e3feb60cd3089f4c6e76f5a6\": rpc error: code = NotFound desc = could not find container \"59300f28a99da412dda92b2752e3d514db7745b1e3feb60cd3089f4c6e76f5a6\": container with ID starting with 59300f28a99da412dda92b2752e3d514db7745b1e3feb60cd3089f4c6e76f5a6 not found: ID does not exist"
Oct 30 22:51:24.316177 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:24.316164    4947 scope.go:122] "RemoveContainer" containerID="5f77232bd34628f0f4bb2344f4a3bb41944c6b722318fc357ad5d82966ba2ba2"
Oct 30 22:51:24.316406 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:51:24.316388    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f77232bd34628f0f4bb2344f4a3bb41944c6b722318fc357ad5d82966ba2ba2\": container with ID starting with 5f77232bd34628f0f4bb2344f4a3bb41944c6b722318fc357ad5d82966ba2ba2 not found: ID does not exist" containerID="5f77232bd34628f0f4bb2344f4a3bb41944c6b722318fc357ad5d82966ba2ba2"
Oct 30 22:51:24.316472 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:24.316412    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f77232bd34628f0f4bb2344f4a3bb41944c6b722318fc357ad5d82966ba2ba2"} err="failed to get container status \"5f77232bd34628f0f4bb2344f4a3bb41944c6b722318fc357ad5d82966ba2ba2\": rpc error: code = NotFound desc = could not find container \"5f77232bd34628f0f4bb2344f4a3bb41944c6b722318fc357ad5d82966ba2ba2\": container with ID starting with 5f77232bd34628f0f4bb2344f4a3bb41944c6b722318fc357ad5d82966ba2ba2 not found: ID does not exist"
Oct 30 22:51:24.511632 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:24.511551    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"]
Oct 30 22:51:24.561493 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:24.561465    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-create-predictor-7d65b89964-d7xbs"]
Oct 30 22:51:26.157660 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:26.157621    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3045563-20c5-4655-b1c1-2d225a590586" path="/var/lib/kubelet/pods/c3045563-20c5-4655-b1c1-2d225a590586/volumes"
Oct 30 22:51:29.048161 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:51:29.048109    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:51:32.197232 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:32.197205    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:51:34.767127 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:34.767097    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"]
Oct 30 22:51:34.767578 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:34.767389    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kserve-container" containerID="cri-o://23033427d653ecdeef1748a4ca36c17c71fe2c2b07cc39ab2d01470f6122dbc1" gracePeriod=30
Oct 30 22:51:34.767578 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:34.767432    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kube-rbac-proxy" containerID="cri-o://0c7d986e517176cd747b8258152b6753daa860588b17966b2a6f13fc5bad3cd0" gracePeriod=30
Oct 30 22:51:35.346945 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:35.346906    4947 generic.go:356] "Generic (PLEG): container finished" podID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerID="0c7d986e517176cd747b8258152b6753daa860588b17966b2a6f13fc5bad3cd0" exitCode=0
Oct 30 22:51:35.347120 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:35.346975    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" event={"ID":"f0f4236b-b86e-467a-b0d4-01139e70d777","Type":"ContainerDied","Data":"0c7d986e517176cd747b8258152b6753daa860588b17966b2a6f13fc5bad3cd0"}
Oct 30 22:51:36.930853 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:36.930804    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Oct 30 22:51:36.936129 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:36.936103    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Oct 30 22:51:39.816055 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.816032    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:51:39.877628 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.877532    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f4236b-b86e-467a-b0d4-01139e70d777-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0f4236b-b86e-467a-b0d4-01139e70d777-kube-rbac-proxy-sar-config\") pod \"f0f4236b-b86e-467a-b0d4-01139e70d777\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") "
Oct 30 22:51:39.877628 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.877599    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0f4236b-b86e-467a-b0d4-01139e70d777-proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0f4236b-b86e-467a-b0d4-01139e70d777-proxy-tls\") pod \"f0f4236b-b86e-467a-b0d4-01139e70d777\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") "
Oct 30 22:51:39.877628 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.877628    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f4236b-b86e-467a-b0d4-01139e70d777-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0f4236b-b86e-467a-b0d4-01139e70d777-kserve-provision-location\") pod \"f0f4236b-b86e-467a-b0d4-01139e70d777\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") "
Oct 30 22:51:39.877893 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.877765    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f4236b-b86e-467a-b0d4-01139e70d777-kube-api-access-wh2w8\" (UniqueName: \"kubernetes.io/projected/f0f4236b-b86e-467a-b0d4-01139e70d777-kube-api-access-wh2w8\") pod \"f0f4236b-b86e-467a-b0d4-01139e70d777\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") "
Oct 30 22:51:39.877893 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.877846    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f4236b-b86e-467a-b0d4-01139e70d777-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0f4236b-b86e-467a-b0d4-01139e70d777-cabundle-cert\") pod \"f0f4236b-b86e-467a-b0d4-01139e70d777\" (UID: \"f0f4236b-b86e-467a-b0d4-01139e70d777\") "
Oct 30 22:51:39.878001 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.877966    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f4236b-b86e-467a-b0d4-01139e70d777-kserve-provision-location" pod "f0f4236b-b86e-467a-b0d4-01139e70d777" (UID: "f0f4236b-b86e-467a-b0d4-01139e70d777"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:39.878001 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.877986    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f4236b-b86e-467a-b0d4-01139e70d777-kube-rbac-proxy-sar-config" pod "f0f4236b-b86e-467a-b0d4-01139e70d777" (UID: "f0f4236b-b86e-467a-b0d4-01139e70d777"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:39.878174 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.878157    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0f4236b-b86e-467a-b0d4-01139e70d777-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:39.878229 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.878175    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0f4236b-b86e-467a-b0d4-01139e70d777-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:39.878229 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.878181    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f4236b-b86e-467a-b0d4-01139e70d777-cabundle-cert" pod "f0f4236b-b86e-467a-b0d4-01139e70d777" (UID: "f0f4236b-b86e-467a-b0d4-01139e70d777"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:39.879995 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.879970    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f4236b-b86e-467a-b0d4-01139e70d777-kube-api-access-wh2w8" pod "f0f4236b-b86e-467a-b0d4-01139e70d777" (UID: "f0f4236b-b86e-467a-b0d4-01139e70d777"). InnerVolumeSpecName "kube-api-access-wh2w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:39.880079 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.879999    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0f4236b-b86e-467a-b0d4-01139e70d777-proxy-tls" pod "f0f4236b-b86e-467a-b0d4-01139e70d777" (UID: "f0f4236b-b86e-467a-b0d4-01139e70d777"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:39.922976 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.922941    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"]
Oct 30 22:51:39.923202 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.923167    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84" podUID="fe522781-a66c-4290-a371-9340c1be0883" containerName="kserve-container" containerID="cri-o://31aa28c9ee572daaec29fc894b4af77c576a525857bf530b943f5dae983e25b4" gracePeriod=30
Oct 30 22:51:39.923320 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.923291    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84" podUID="fe522781-a66c-4290-a371-9340c1be0883" containerName="kube-rbac-proxy" containerID="cri-o://c143c8e37e9ea1f67f1d77c9a6b8eea6381481fd233b89770ad3ba18938bc045" gracePeriod=30
Oct 30 22:51:39.979432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.979388    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh2w8\" (UniqueName: \"kubernetes.io/projected/f0f4236b-b86e-467a-b0d4-01139e70d777-kube-api-access-wh2w8\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:39.979432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.979428    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0f4236b-b86e-467a-b0d4-01139e70d777-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:39.979432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:39.979438    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0f4236b-b86e-467a-b0d4-01139e70d777-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:40.375900 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:40.375867    4947 generic.go:356] "Generic (PLEG): container finished" podID="fe522781-a66c-4290-a371-9340c1be0883" containerID="c143c8e37e9ea1f67f1d77c9a6b8eea6381481fd233b89770ad3ba18938bc045" exitCode=0
Oct 30 22:51:40.376078 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:40.375941    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84" event={"ID":"fe522781-a66c-4290-a371-9340c1be0883","Type":"ContainerDied","Data":"c143c8e37e9ea1f67f1d77c9a6b8eea6381481fd233b89770ad3ba18938bc045"}
Oct 30 22:51:40.377705 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:40.377686    4947 generic.go:356] "Generic (PLEG): container finished" podID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerID="23033427d653ecdeef1748a4ca36c17c71fe2c2b07cc39ab2d01470f6122dbc1" exitCode=0
Oct 30 22:51:40.377823 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:40.377723    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" event={"ID":"f0f4236b-b86e-467a-b0d4-01139e70d777","Type":"ContainerDied","Data":"23033427d653ecdeef1748a4ca36c17c71fe2c2b07cc39ab2d01470f6122dbc1"}
Oct 30 22:51:40.377823 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:40.377747    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"
Oct 30 22:51:40.377823 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:40.377761    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv" event={"ID":"f0f4236b-b86e-467a-b0d4-01139e70d777","Type":"ContainerDied","Data":"2132a71ee030ea0182c472d03db3595999c879b0f407961bd55000215400db58"}
Oct 30 22:51:40.377823 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:40.377786    4947 scope.go:122] "RemoveContainer" containerID="0c7d986e517176cd747b8258152b6753daa860588b17966b2a6f13fc5bad3cd0"
Oct 30 22:51:40.387518 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:40.387497    4947 scope.go:122] "RemoveContainer" containerID="23033427d653ecdeef1748a4ca36c17c71fe2c2b07cc39ab2d01470f6122dbc1"
Oct 30 22:51:40.395026 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:40.395008    4947 scope.go:122] "RemoveContainer" containerID="96ac1e99480bc6d8a56e9fecd5122a8f944e69b27a5494d3bf4d5add68305e0c"
Oct 30 22:51:40.402312 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:40.402292    4947 scope.go:122] "RemoveContainer" containerID="0c7d986e517176cd747b8258152b6753daa860588b17966b2a6f13fc5bad3cd0"
Oct 30 22:51:40.402639 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:51:40.402615    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c7d986e517176cd747b8258152b6753daa860588b17966b2a6f13fc5bad3cd0\": container with ID starting with 0c7d986e517176cd747b8258152b6753daa860588b17966b2a6f13fc5bad3cd0 not found: ID does not exist" containerID="0c7d986e517176cd747b8258152b6753daa860588b17966b2a6f13fc5bad3cd0"
Oct 30 22:51:40.402704 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:40.402650    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7d986e517176cd747b8258152b6753daa860588b17966b2a6f13fc5bad3cd0"} err="failed to get container status \"0c7d986e517176cd747b8258152b6753daa860588b17966b2a6f13fc5bad3cd0\": rpc error: code = NotFound desc = could not find container \"0c7d986e517176cd747b8258152b6753daa860588b17966b2a6f13fc5bad3cd0\": container with ID starting with 0c7d986e517176cd747b8258152b6753daa860588b17966b2a6f13fc5bad3cd0 not found: ID does not exist"
Oct 30 22:51:40.402704 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:40.402670    4947 scope.go:122] "RemoveContainer" containerID="23033427d653ecdeef1748a4ca36c17c71fe2c2b07cc39ab2d01470f6122dbc1"
Oct 30 22:51:40.402908 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:51:40.402889    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23033427d653ecdeef1748a4ca36c17c71fe2c2b07cc39ab2d01470f6122dbc1\": container with ID starting with 23033427d653ecdeef1748a4ca36c17c71fe2c2b07cc39ab2d01470f6122dbc1 not found: ID does not exist" containerID="23033427d653ecdeef1748a4ca36c17c71fe2c2b07cc39ab2d01470f6122dbc1"
Oct 30 22:51:40.402962 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:40.402913    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23033427d653ecdeef1748a4ca36c17c71fe2c2b07cc39ab2d01470f6122dbc1"} err="failed to get container status \"23033427d653ecdeef1748a4ca36c17c71fe2c2b07cc39ab2d01470f6122dbc1\": rpc error: code = NotFound desc = could not find container \"23033427d653ecdeef1748a4ca36c17c71fe2c2b07cc39ab2d01470f6122dbc1\": container with ID starting with 23033427d653ecdeef1748a4ca36c17c71fe2c2b07cc39ab2d01470f6122dbc1 not found: ID does not exist"
Oct 30 22:51:40.402962 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:40.402926    4947 scope.go:122] "RemoveContainer" containerID="96ac1e99480bc6d8a56e9fecd5122a8f944e69b27a5494d3bf4d5add68305e0c"
Oct 30 22:51:40.403167 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:51:40.403151    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ac1e99480bc6d8a56e9fecd5122a8f944e69b27a5494d3bf4d5add68305e0c\": container with ID starting with 96ac1e99480bc6d8a56e9fecd5122a8f944e69b27a5494d3bf4d5add68305e0c not found: ID does not exist" containerID="96ac1e99480bc6d8a56e9fecd5122a8f944e69b27a5494d3bf4d5add68305e0c"
Oct 30 22:51:40.403220 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:40.403172    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ac1e99480bc6d8a56e9fecd5122a8f944e69b27a5494d3bf4d5add68305e0c"} err="failed to get container status \"96ac1e99480bc6d8a56e9fecd5122a8f944e69b27a5494d3bf4d5add68305e0c\": rpc error: code = NotFound desc = could not find container \"96ac1e99480bc6d8a56e9fecd5122a8f944e69b27a5494d3bf4d5add68305e0c\": container with ID starting with 96ac1e99480bc6d8a56e9fecd5122a8f944e69b27a5494d3bf4d5add68305e0c not found: ID does not exist"
Oct 30 22:51:40.546433 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:40.546400    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"]
Oct 30 22:51:40.596136 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:40.596109    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-create-v2-predictor-76d6db7cf9-r28xv"]
Oct 30 22:51:42.158051 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:42.158013    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" path="/var/lib/kubelet/pods/f0f4236b-b86e-467a-b0d4-01139e70d777/volumes"
Oct 30 22:51:42.175561 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:42.175519    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84" podUID="fe522781-a66c-4290-a371-9340c1be0883" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Oct 30 22:51:43.831051 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.831018    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"]
Oct 30 22:51:43.831432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.831361    4947 cpu_manager.go:414] "containerMap: removing container" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="storage-initializer"
Oct 30 22:51:43.831432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.831372    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="storage-initializer"
Oct 30 22:51:43.831432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.831384    4947 cpu_manager.go:414] "containerMap: removing container" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kserve-container"
Oct 30 22:51:43.831432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.831389    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kserve-container"
Oct 30 22:51:43.831432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.831397    4947 cpu_manager.go:414] "containerMap: removing container" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kube-rbac-proxy"
Oct 30 22:51:43.831432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.831402    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kube-rbac-proxy"
Oct 30 22:51:43.831432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.831412    4947 cpu_manager.go:414] "containerMap: removing container" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kube-rbac-proxy"
Oct 30 22:51:43.831432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.831416    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kube-rbac-proxy"
Oct 30 22:51:43.831432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.831428    4947 cpu_manager.go:414] "containerMap: removing container" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kserve-container"
Oct 30 22:51:43.831432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.831432    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kserve-container"
Oct 30 22:51:43.831736 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.831443    4947 cpu_manager.go:414] "containerMap: removing container" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="storage-initializer"
Oct 30 22:51:43.831736 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.831448    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="storage-initializer"
Oct 30 22:51:43.831736 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.831504    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kserve-container"
Oct 30 22:51:43.831736 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.831512    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kube-rbac-proxy"
Oct 30 22:51:43.831736 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.831518    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f4236b-b86e-467a-b0d4-01139e70d777" containerName="kube-rbac-proxy"
Oct 30 22:51:43.831736 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.831524    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3045563-20c5-4655-b1c1-2d225a590586" containerName="kserve-container"
Oct 30 22:51:43.834307 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.834290    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:43.888229 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.888195    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"seaweedfs-s3-creds\""
Oct 30 22:51:43.888396 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.888381    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"default-dockercfg-s9cqw\""
Oct 30 22:51:43.910828 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.910796    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-66ffb95d77-2s76g\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:43.910951 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.910892    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-66ffb95d77-2s76g\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:43.911004 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.910947    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c79qq\" (UniqueName: \"kubernetes.io/projected/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kube-api-access-c79qq\") pod \"isvc-canary-promote-v2-predictor-66ffb95d77-2s76g\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:43.911004 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.910968    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-66ffb95d77-2s76g\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:43.911004 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.910988    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-66ffb95d77-2s76g\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:43.937695 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.937661    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Oct 30 22:51:43.937695 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.937670    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"openshift-service-ca.crt\""
Oct 30 22:51:43.937920 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.937814    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Oct 30 22:51:43.937920 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.937835    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"kube-root-ca.crt\""
Oct 30 22:51:43.937920 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.937814    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:51:43.939999 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:43.939974    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"]
Oct 30 22:51:44.012116 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.012076    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-66ffb95d77-2s76g\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:44.012292 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.012143    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-66ffb95d77-2s76g\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:44.012292 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.012163    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c79qq\" (UniqueName: \"kubernetes.io/projected/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kube-api-access-c79qq\") pod \"isvc-canary-promote-v2-predictor-66ffb95d77-2s76g\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:44.012292 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.012182    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-66ffb95d77-2s76g\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:44.012292 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.012201    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-66ffb95d77-2s76g\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:44.012541 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.012495    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-66ffb95d77-2s76g\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:44.012863 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.012842    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-66ffb95d77-2s76g\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:44.012974 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.012955    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-66ffb95d77-2s76g\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:44.014905 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.014886    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-66ffb95d77-2s76g\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:44.075094 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.075052    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c79qq\" (UniqueName: \"kubernetes.io/projected/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kube-api-access-c79qq\") pod \"isvc-canary-promote-v2-predictor-66ffb95d77-2s76g\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:44.147189 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.147099    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:44.573011 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.572977    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"]
Oct 30 22:51:44.576469 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:51:44.576439    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fdc4ce5_a233_4a9c_9767_f771db1cb82c.slice/crio-8870f39b5e0f03d9d64797cce34f4ba52ce651e4686f058eac2ea89491121b74 WatchSource:0}: Error finding container 8870f39b5e0f03d9d64797cce34f4ba52ce651e4686f058eac2ea89491121b74: Status 404 returned error can't find the container with id 8870f39b5e0f03d9d64797cce34f4ba52ce651e4686f058eac2ea89491121b74
Oct 30 22:51:44.823880 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.823793    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"]
Oct 30 22:51:44.826344 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.826316    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:44.880169 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.880140    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-canary-promote-predictor-serving-cert\""
Oct 30 22:51:44.889895 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.889866    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"]
Oct 30 22:51:44.920579 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.920544    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-66d68c5b56-tn972\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:44.920719 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.920591    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kserve-provision-location\") pod \"isvc-canary-promote-predictor-66d68c5b56-tn972\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:44.920796 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.920766    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q72qb\" (UniqueName: \"kubernetes.io/projected/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kube-api-access-q72qb\") pod \"isvc-canary-promote-predictor-66d68c5b56-tn972\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:44.920851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.920815    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b8acec7-a35b-41a2-b201-8d2a37fbb439-proxy-tls\") pod \"isvc-canary-promote-predictor-66d68c5b56-tn972\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:44.920885 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:44.920845    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b8acec7-a35b-41a2-b201-8d2a37fbb439-cabundle-cert\") pod \"isvc-canary-promote-predictor-66d68c5b56-tn972\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:45.021936 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:45.021904    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q72qb\" (UniqueName: \"kubernetes.io/projected/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kube-api-access-q72qb\") pod \"isvc-canary-promote-predictor-66d68c5b56-tn972\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:45.022095 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:45.021946    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b8acec7-a35b-41a2-b201-8d2a37fbb439-proxy-tls\") pod \"isvc-canary-promote-predictor-66d68c5b56-tn972\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:45.022095 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:45.021975    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b8acec7-a35b-41a2-b201-8d2a37fbb439-cabundle-cert\") pod \"isvc-canary-promote-predictor-66d68c5b56-tn972\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:45.022219 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:45.022146    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-66d68c5b56-tn972\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:45.022219 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:45.022193    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kserve-provision-location\") pod \"isvc-canary-promote-predictor-66d68c5b56-tn972\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:45.022617 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:45.022593    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kserve-provision-location\") pod \"isvc-canary-promote-predictor-66d68c5b56-tn972\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:45.022736 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:45.022726    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b8acec7-a35b-41a2-b201-8d2a37fbb439-cabundle-cert\") pod \"isvc-canary-promote-predictor-66d68c5b56-tn972\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:45.022942 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:45.022920    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-66d68c5b56-tn972\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:45.024671 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:45.024652    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b8acec7-a35b-41a2-b201-8d2a37fbb439-proxy-tls\") pod \"isvc-canary-promote-predictor-66d68c5b56-tn972\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:45.085307 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:45.085226    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q72qb\" (UniqueName: \"kubernetes.io/projected/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kube-api-access-q72qb\") pod \"isvc-canary-promote-predictor-66d68c5b56-tn972\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") " pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:45.139543 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:45.139505    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:45.402011 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:45.401929    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" event={"ID":"6fdc4ce5-a233-4a9c-9767-f771db1cb82c","Type":"ContainerStarted","Data":"a9ed98562ff9b2e4c96b5fd5e245cb2a5487a3bf6714bba02d6d4368c862e875"}
Oct 30 22:51:45.402011 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:45.401988    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" event={"ID":"6fdc4ce5-a233-4a9c-9767-f771db1cb82c","Type":"ContainerStarted","Data":"8870f39b5e0f03d9d64797cce34f4ba52ce651e4686f058eac2ea89491121b74"}
Oct 30 22:51:45.568486 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:51:45.568460    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8acec7_a35b_41a2_b201_8d2a37fbb439.slice/crio-4f9ae05f6582a3fed480e6f16508bee199a1ea47b80dd6f55e707cc7a863fea0 WatchSource:0}: Error finding container 4f9ae05f6582a3fed480e6f16508bee199a1ea47b80dd6f55e707cc7a863fea0: Status 404 returned error can't find the container with id 4f9ae05f6582a3fed480e6f16508bee199a1ea47b80dd6f55e707cc7a863fea0
Oct 30 22:51:45.578188 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:45.578163    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"]
Oct 30 22:51:46.409045 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.409011    4947 generic.go:356] "Generic (PLEG): container finished" podID="fe522781-a66c-4290-a371-9340c1be0883" containerID="31aa28c9ee572daaec29fc894b4af77c576a525857bf530b943f5dae983e25b4" exitCode=0
Oct 30 22:51:46.409459 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.409088    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84" event={"ID":"fe522781-a66c-4290-a371-9340c1be0883","Type":"ContainerDied","Data":"31aa28c9ee572daaec29fc894b4af77c576a525857bf530b943f5dae983e25b4"}
Oct 30 22:51:46.417191 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.417118    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" event={"ID":"9b8acec7-a35b-41a2-b201-8d2a37fbb439","Type":"ContainerStarted","Data":"10a31539dfa6d0b8e3ae1bc2034f613de791c5c802de3357204bfbcc724c9831"}
Oct 30 22:51:46.417408 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.417257    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" event={"ID":"9b8acec7-a35b-41a2-b201-8d2a37fbb439","Type":"ContainerStarted","Data":"4f9ae05f6582a3fed480e6f16508bee199a1ea47b80dd6f55e707cc7a863fea0"}
Oct 30 22:51:46.465466 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.465444    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:51:46.533403 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.533363    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe522781-a66c-4290-a371-9340c1be0883-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe522781-a66c-4290-a371-9340c1be0883-kube-rbac-proxy-sar-config\") pod \"fe522781-a66c-4290-a371-9340c1be0883\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") "
Oct 30 22:51:46.533538 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.533414    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe522781-a66c-4290-a371-9340c1be0883-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe522781-a66c-4290-a371-9340c1be0883-cabundle-cert\") pod \"fe522781-a66c-4290-a371-9340c1be0883\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") "
Oct 30 22:51:46.533538 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.533517    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe522781-a66c-4290-a371-9340c1be0883-kube-api-access-j5zhw\" (UniqueName: \"kubernetes.io/projected/fe522781-a66c-4290-a371-9340c1be0883-kube-api-access-j5zhw\") pod \"fe522781-a66c-4290-a371-9340c1be0883\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") "
Oct 30 22:51:46.533645 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.533551    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe522781-a66c-4290-a371-9340c1be0883-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe522781-a66c-4290-a371-9340c1be0883-kserve-provision-location\") pod \"fe522781-a66c-4290-a371-9340c1be0883\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") "
Oct 30 22:51:46.533711 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.533648    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe522781-a66c-4290-a371-9340c1be0883-proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe522781-a66c-4290-a371-9340c1be0883-proxy-tls\") pod \"fe522781-a66c-4290-a371-9340c1be0883\" (UID: \"fe522781-a66c-4290-a371-9340c1be0883\") "
Oct 30 22:51:46.533799 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.533777    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe522781-a66c-4290-a371-9340c1be0883-kube-rbac-proxy-sar-config" pod "fe522781-a66c-4290-a371-9340c1be0883" (UID: "fe522781-a66c-4290-a371-9340c1be0883"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:46.533842 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.533825    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe522781-a66c-4290-a371-9340c1be0883-cabundle-cert" pod "fe522781-a66c-4290-a371-9340c1be0883" (UID: "fe522781-a66c-4290-a371-9340c1be0883"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:46.533883 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.533835    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe522781-a66c-4290-a371-9340c1be0883-kserve-provision-location" pod "fe522781-a66c-4290-a371-9340c1be0883" (UID: "fe522781-a66c-4290-a371-9340c1be0883"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:46.533934 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.533915    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe522781-a66c-4290-a371-9340c1be0883-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:46.533971 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.533942    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe522781-a66c-4290-a371-9340c1be0883-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:46.533971 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.533959    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe522781-a66c-4290-a371-9340c1be0883-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:46.535740 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.535721    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe522781-a66c-4290-a371-9340c1be0883-proxy-tls" pod "fe522781-a66c-4290-a371-9340c1be0883" (UID: "fe522781-a66c-4290-a371-9340c1be0883"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:46.535834 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.535774    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe522781-a66c-4290-a371-9340c1be0883-kube-api-access-j5zhw" pod "fe522781-a66c-4290-a371-9340c1be0883" (UID: "fe522781-a66c-4290-a371-9340c1be0883"). InnerVolumeSpecName "kube-api-access-j5zhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:46.634508 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.634440    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5zhw\" (UniqueName: \"kubernetes.io/projected/fe522781-a66c-4290-a371-9340c1be0883-kube-api-access-j5zhw\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:46.634508 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:46.634466    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe522781-a66c-4290-a371-9340c1be0883-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:47.417860 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:47.417826    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"
Oct 30 22:51:47.417860 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:47.417855    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84" event={"ID":"fe522781-a66c-4290-a371-9340c1be0883","Type":"ContainerDied","Data":"8ab66aa17ac4db2d67bdd5cff53c1ac9d1c3223f6303e703f5fdccde9b97a241"}
Oct 30 22:51:47.418411 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:47.417917    4947 scope.go:122] "RemoveContainer" containerID="c143c8e37e9ea1f67f1d77c9a6b8eea6381481fd233b89770ad3ba18938bc045"
Oct 30 22:51:47.426829 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:47.426807    4947 scope.go:122] "RemoveContainer" containerID="31aa28c9ee572daaec29fc894b4af77c576a525857bf530b943f5dae983e25b4"
Oct 30 22:51:47.434371 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:47.434356    4947 scope.go:122] "RemoveContainer" containerID="30d4bab49f9d3357114e4dfb8565ff3a6705a9d46973c3cb53feea259272b197"
Oct 30 22:51:47.455394 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:51:47.455366    4947 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe522781_a66c_4290_a371_9340c1be0883.slice/crio-8ab66aa17ac4db2d67bdd5cff53c1ac9d1c3223f6303e703f5fdccde9b97a241\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe522781_a66c_4290_a371_9340c1be0883.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:51:47.588404 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:47.588371    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"]
Oct 30 22:51:47.594522 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:47.594496    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/xgboost-v2-mlserver-predictor-84697fb8cb-tbn84"]
Oct 30 22:51:48.158492 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:48.158460    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe522781-a66c-4290-a371-9340c1be0883" path="/var/lib/kubelet/pods/fe522781-a66c-4290-a371-9340c1be0883/volumes"
Oct 30 22:51:49.429921 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:49.429884    4947 generic.go:356] "Generic (PLEG): container finished" podID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerID="10a31539dfa6d0b8e3ae1bc2034f613de791c5c802de3357204bfbcc724c9831" exitCode=0
Oct 30 22:51:49.430382 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:49.429959    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" event={"ID":"9b8acec7-a35b-41a2-b201-8d2a37fbb439","Type":"ContainerDied","Data":"10a31539dfa6d0b8e3ae1bc2034f613de791c5c802de3357204bfbcc724c9831"}
Oct 30 22:51:49.431655 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:49.431635    4947 generic.go:356] "Generic (PLEG): container finished" podID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerID="a9ed98562ff9b2e4c96b5fd5e245cb2a5487a3bf6714bba02d6d4368c862e875" exitCode=0
Oct 30 22:51:49.431755 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:49.431670    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" event={"ID":"6fdc4ce5-a233-4a9c-9767-f771db1cb82c","Type":"ContainerDied","Data":"a9ed98562ff9b2e4c96b5fd5e245cb2a5487a3bf6714bba02d6d4368c862e875"}
Oct 30 22:51:50.438638 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:50.438601    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" event={"ID":"9b8acec7-a35b-41a2-b201-8d2a37fbb439","Type":"ContainerStarted","Data":"92b5a55b9a7bf274164d15b0a790e63a313826fffd6feb53884a8492ed43ea95"}
Oct 30 22:51:50.439099 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:50.438701    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" event={"ID":"9b8acec7-a35b-41a2-b201-8d2a37fbb439","Type":"ContainerStarted","Data":"90d3ff15f261cb4f573638df8da532e573a1d1bab77d8e0fd048664c7dbf358c"}
Oct 30 22:51:50.439099 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:50.439064    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:50.439292 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:50.439275    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:50.441535 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:50.441495    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 22:51:50.441623 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:50.441574    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" event={"ID":"6fdc4ce5-a233-4a9c-9767-f771db1cb82c","Type":"ContainerStarted","Data":"88b7c2d5e0a32bbe315209d0b82e497f417a372610d09697c361097d89fea532"}
Oct 30 22:51:50.441678 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:50.441628    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" event={"ID":"6fdc4ce5-a233-4a9c-9767-f771db1cb82c","Type":"ContainerStarted","Data":"67409ee9207c4007ed4ab8d4d1651aefa2071a842a7890bbf005f41c9bbcb777"}
Oct 30 22:51:50.441895 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:50.441878    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:50.441961 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:50.441926    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:50.443048 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:50.443020    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Oct 30 22:51:50.560638 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:50.560583    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" podStartSLOduration=2.852885134 podStartE2EDuration="6.560569792s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.707684658s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:50.558978535 +0000 UTC m=+2608.558326099" watchObservedRunningTime="2026-10-30 22:51:50.560569792 +0000 UTC m=+2608.559917355"
Oct 30 22:51:50.681160 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:50.681106    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" podStartSLOduration=3.8377719519999998 podStartE2EDuration="7.681088627s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.843316675s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:50.68061544 +0000 UTC m=+2608.679963023" watchObservedRunningTime="2026-10-30 22:51:50.681088627 +0000 UTC m=+2608.680436191"
Oct 30 22:51:51.019203 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.019168    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"]
Oct 30 22:51:51.019525 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.019512    4947 cpu_manager.go:414] "containerMap: removing container" podUID="fe522781-a66c-4290-a371-9340c1be0883" containerName="kube-rbac-proxy"
Oct 30 22:51:51.019590 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.019526    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe522781-a66c-4290-a371-9340c1be0883" containerName="kube-rbac-proxy"
Oct 30 22:51:51.019590 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.019539    4947 cpu_manager.go:414] "containerMap: removing container" podUID="fe522781-a66c-4290-a371-9340c1be0883" containerName="kserve-container"
Oct 30 22:51:51.019590 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.019544    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe522781-a66c-4290-a371-9340c1be0883" containerName="kserve-container"
Oct 30 22:51:51.019590 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.019573    4947 cpu_manager.go:414] "containerMap: removing container" podUID="fe522781-a66c-4290-a371-9340c1be0883" containerName="storage-initializer"
Oct 30 22:51:51.019590 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.019578    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe522781-a66c-4290-a371-9340c1be0883" containerName="storage-initializer"
Oct 30 22:51:51.019763 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.019632    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe522781-a66c-4290-a371-9340c1be0883" containerName="kube-rbac-proxy"
Oct 30 22:51:51.019763 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.019641    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe522781-a66c-4290-a371-9340c1be0883" containerName="kserve-container"
Oct 30 22:51:51.022792 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.022776    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:51.072214 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.072178    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:51.072405 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.072221    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:51.072405 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.072296    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df89h\" (UniqueName: \"kubernetes.io/projected/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kube-api-access-df89h\") pod \"isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:51.072543 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.072445    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:51.072543 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.072482    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:51.076835 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.076816    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:51:51.076942 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.076931    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:51:51.076996 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.076940    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Oct 30 22:51:51.077554 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.077538    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:51:51.077554 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.077546    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Oct 30 22:51:51.077665 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.077556    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:51:51.077665 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.077563    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:51:51.084023 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.083990    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"]
Oct 30 22:51:51.173436 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.173406    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:51.173436 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.173439    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:51.173672 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.173477    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:51.173672 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.173497    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:51.173672 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.173617    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df89h\" (UniqueName: \"kubernetes.io/projected/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kube-api-access-df89h\") pod \"isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:51.173940 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.173915    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:51.174169 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.174150    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:51.174207 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.174186    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:51.176062 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.176043    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:51.236726 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.236695    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df89h\" (UniqueName: \"kubernetes.io/projected/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kube-api-access-df89h\") pod \"isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:51.339883 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.339839    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:51.447307 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.447268    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Oct 30 22:51:51.447718 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.447439    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 22:51:51.765647 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:51.765572    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"]
Oct 30 22:51:51.768612 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:51:51.768571    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e80d60_a2d2_4ef9_ba55_8b2bfc44da96.slice/crio-2562beaeab37f0868e850987085eb42fda39979faaf5666f0fb21934d6fbe015 WatchSource:0}: Error finding container 2562beaeab37f0868e850987085eb42fda39979faaf5666f0fb21934d6fbe015: Status 404 returned error can't find the container with id 2562beaeab37f0868e850987085eb42fda39979faaf5666f0fb21934d6fbe015
Oct 30 22:51:52.454413 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:52.454367    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" event={"ID":"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96","Type":"ContainerStarted","Data":"691ee5e2dead4bdd8cd08618e5d413963b3d34c8403df3d3345b4d319b1abb8a"}
Oct 30 22:51:52.454887 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:52.454427    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" event={"ID":"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96","Type":"ContainerStarted","Data":"2562beaeab37f0868e850987085eb42fda39979faaf5666f0fb21934d6fbe015"}
Oct 30 22:51:56.452147 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:56.452114    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:51:56.452608 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:56.452519    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:51:56.452663 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:56.452633    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 22:51:56.453038 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:56.453017    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Oct 30 22:51:56.472350 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:56.472306    4947 generic.go:356] "Generic (PLEG): container finished" podID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerID="691ee5e2dead4bdd8cd08618e5d413963b3d34c8403df3d3345b4d319b1abb8a" exitCode=0
Oct 30 22:51:56.472487 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:56.472379    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" event={"ID":"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96","Type":"ContainerDied","Data":"691ee5e2dead4bdd8cd08618e5d413963b3d34c8403df3d3345b4d319b1abb8a"}
Oct 30 22:51:57.479432 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:57.479393    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" event={"ID":"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96","Type":"ContainerStarted","Data":"337b24f1f19f4b801915fe151e6ffabe256855a5871bdefc73ddfa38e4a681ef"}
Oct 30 22:51:57.479822 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:57.479463    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" event={"ID":"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96","Type":"ContainerStarted","Data":"52d9516c5ef55ae45ecfa2ccdab07bf0faf278b44d2191073b5538380ed15cdf"}
Oct 30 22:51:57.479822 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:57.479732    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:57.479931 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:57.479875    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:51:57.480977 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:57.480953    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 22:51:57.601490 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:57.601422    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" podStartSLOduration=3.52143487 podStartE2EDuration="7.601402836s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.079967966s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:57.600548739 +0000 UTC m=+2615.599896305" watchObservedRunningTime="2026-10-30 22:51:57.601402836 +0000 UTC m=+2615.600750400"
Oct 30 22:51:58.484150 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:51:58.484114    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 22:51:59.047856 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:51:59.047817    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:52:03.488885 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:03.488855    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:52:03.489519 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:03.489489    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 22:52:06.453128 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:06.453095    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 22:52:06.453523 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:06.453093    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Oct 30 22:52:13.490080 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:13.490041    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 22:52:16.453101 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:16.453056    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 22:52:16.453516 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:16.453056    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Oct 30 22:52:23.489981 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:23.489944    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 22:52:26.452672 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:26.452633    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 22:52:26.453164 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:26.452940    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Oct 30 22:52:29.048199 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:52:29.048162    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:52:33.489790 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:33.489704    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 22:52:36.452769 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:36.452732    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 22:52:36.453220 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:36.452980    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Oct 30 22:52:43.489669 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:43.489629    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 22:52:46.452669 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:46.452634    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 22:52:46.453082 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:46.452903    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Oct 30 22:52:53.489878 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:53.489829    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 22:52:56.452570 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:56.452534    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 22:52:56.452967 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:52:56.452873    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Oct 30 22:52:59.048670 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:52:59.048635    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:53:01.153658 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:01.153614    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Oct 30 22:53:03.489906 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:03.489879    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:53:06.453676 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:06.453648    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:53:11.154062 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:11.154032    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:53:11.639985 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:11.639952    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"]
Oct 30 22:53:11.640463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:11.640199    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kserve-container" containerID="cri-o://52d9516c5ef55ae45ecfa2ccdab07bf0faf278b44d2191073b5538380ed15cdf" gracePeriod=30
Oct 30 22:53:11.640463 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:11.640247    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kube-rbac-proxy" containerID="cri-o://337b24f1f19f4b801915fe151e6ffabe256855a5871bdefc73ddfa38e4a681ef" gracePeriod=30
Oct 30 22:53:11.802382 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:11.802352    4947 generic.go:356] "Generic (PLEG): container finished" podID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerID="337b24f1f19f4b801915fe151e6ffabe256855a5871bdefc73ddfa38e4a681ef" exitCode=0
Oct 30 22:53:11.802538 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:11.802426    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" event={"ID":"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96","Type":"ContainerDied","Data":"337b24f1f19f4b801915fe151e6ffabe256855a5871bdefc73ddfa38e4a681ef"}
Oct 30 22:53:13.485293 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:13.485247    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.63:8643/healthz\": dial tcp 10.132.0.63:8643: connect: connection refused"
Oct 30 22:53:13.489528 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:13.489504    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Oct 30 22:53:13.591072 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:13.591040    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"]
Oct 30 22:53:13.591354 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:13.591304    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kserve-container" containerID="cri-o://90d3ff15f261cb4f573638df8da532e573a1d1bab77d8e0fd048664c7dbf358c" gracePeriod=30
Oct 30 22:53:13.591446 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:13.591390    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kube-rbac-proxy" containerID="cri-o://92b5a55b9a7bf274164d15b0a790e63a313826fffd6feb53884a8492ed43ea95" gracePeriod=30
Oct 30 22:53:13.814125 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:13.814083    4947 generic.go:356] "Generic (PLEG): container finished" podID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerID="92b5a55b9a7bf274164d15b0a790e63a313826fffd6feb53884a8492ed43ea95" exitCode=0
Oct 30 22:53:13.814319 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:13.814208    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" event={"ID":"9b8acec7-a35b-41a2-b201-8d2a37fbb439","Type":"ContainerDied","Data":"92b5a55b9a7bf274164d15b0a790e63a313826fffd6feb53884a8492ed43ea95"}
Oct 30 22:53:15.825981 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:15.825948    4947 generic.go:356] "Generic (PLEG): container finished" podID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerID="52d9516c5ef55ae45ecfa2ccdab07bf0faf278b44d2191073b5538380ed15cdf" exitCode=0
Oct 30 22:53:15.826361 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:15.826011    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" event={"ID":"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96","Type":"ContainerDied","Data":"52d9516c5ef55ae45ecfa2ccdab07bf0faf278b44d2191073b5538380ed15cdf"}
Oct 30 22:53:15.881840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:15.881815    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:53:15.991632 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:15.991551    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kube-rbac-proxy-sar-config\") pod \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") "
Oct 30 22:53:15.991632 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:15.991604    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kserve-provision-location\") pod \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") "
Oct 30 22:53:15.991861 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:15.991640    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-cabundle-cert\") pod \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") "
Oct 30 22:53:15.991861 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:15.991697    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kube-api-access-df89h\" (UniqueName: \"kubernetes.io/projected/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kube-api-access-df89h\") pod \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") "
Oct 30 22:53:15.991861 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:15.991736    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-proxy-tls\" (UniqueName: \"kubernetes.io/secret/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-proxy-tls\") pod \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\" (UID: \"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96\") "
Oct 30 22:53:15.992012 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:15.991936    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kube-rbac-proxy-sar-config" pod "25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" (UID: "25e80d60-a2d2-4ef9-ba55-8b2bfc44da96"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:15.992012 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:15.991980    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kserve-provision-location" pod "25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" (UID: "25e80d60-a2d2-4ef9-ba55-8b2bfc44da96"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:15.992083 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:15.992022    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-cabundle-cert" pod "25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" (UID: "25e80d60-a2d2-4ef9-ba55-8b2bfc44da96"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:15.993947 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:15.993925    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kube-api-access-df89h" pod "25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" (UID: "25e80d60-a2d2-4ef9-ba55-8b2bfc44da96"). InnerVolumeSpecName "kube-api-access-df89h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:15.994068 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:15.993957    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-proxy-tls" pod "25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" (UID: "25e80d60-a2d2-4ef9-ba55-8b2bfc44da96"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:16.092607 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:16.092580    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:16.092607 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:16.092604    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df89h\" (UniqueName: \"kubernetes.io/projected/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kube-api-access-df89h\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:16.092759 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:16.092615    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:16.092759 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:16.092624    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:16.092759 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:16.092633    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:16.447744 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:16.447700    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Oct 30 22:53:16.453413 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:16.453385    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 30 22:53:16.832506 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:16.832477    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"
Oct 30 22:53:16.832506 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:16.832490    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx" event={"ID":"25e80d60-a2d2-4ef9-ba55-8b2bfc44da96","Type":"ContainerDied","Data":"2562beaeab37f0868e850987085eb42fda39979faaf5666f0fb21934d6fbe015"}
Oct 30 22:53:16.833030 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:16.832566    4947 scope.go:122] "RemoveContainer" containerID="337b24f1f19f4b801915fe151e6ffabe256855a5871bdefc73ddfa38e4a681ef"
Oct 30 22:53:16.841165 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:16.841140    4947 scope.go:122] "RemoveContainer" containerID="52d9516c5ef55ae45ecfa2ccdab07bf0faf278b44d2191073b5538380ed15cdf"
Oct 30 22:53:16.848677 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:16.848655    4947 scope.go:122] "RemoveContainer" containerID="691ee5e2dead4bdd8cd08618e5d413963b3d34c8403df3d3345b4d319b1abb8a"
Oct 30 22:53:17.000371 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:17.000321    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"]
Oct 30 22:53:17.049968 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:17.049933    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-runtime-predictor-7f578f55f9-8n7rx"]
Oct 30 22:53:18.163172 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.163144    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" path="/var/lib/kubelet/pods/25e80d60-a2d2-4ef9-ba55-8b2bfc44da96/volumes"
Oct 30 22:53:18.331948 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.331920    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:53:18.413647 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.413576    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kube-api-access-q72qb\" (UniqueName: \"kubernetes.io/projected/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kube-api-access-q72qb\") pod \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") "
Oct 30 22:53:18.413647 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.413620    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b8acec7-a35b-41a2-b201-8d2a37fbb439-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b8acec7-a35b-41a2-b201-8d2a37fbb439-cabundle-cert\") pod \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") "
Oct 30 22:53:18.413647 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.413642    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b8acec7-a35b-41a2-b201-8d2a37fbb439-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b8acec7-a35b-41a2-b201-8d2a37fbb439-proxy-tls\") pod \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") "
Oct 30 22:53:18.413910 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.413682    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kube-rbac-proxy-sar-config\") pod \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") "
Oct 30 22:53:18.413910 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.413700    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kserve-provision-location\") pod \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\" (UID: \"9b8acec7-a35b-41a2-b201-8d2a37fbb439\") "
Oct 30 22:53:18.414090 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.414059    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8acec7-a35b-41a2-b201-8d2a37fbb439-cabundle-cert" pod "9b8acec7-a35b-41a2-b201-8d2a37fbb439" (UID: "9b8acec7-a35b-41a2-b201-8d2a37fbb439"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:18.414090 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.414059    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kserve-provision-location" pod "9b8acec7-a35b-41a2-b201-8d2a37fbb439" (UID: "9b8acec7-a35b-41a2-b201-8d2a37fbb439"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:18.414451 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.414150    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kube-rbac-proxy-sar-config" pod "9b8acec7-a35b-41a2-b201-8d2a37fbb439" (UID: "9b8acec7-a35b-41a2-b201-8d2a37fbb439"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:18.415898 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.415881    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kube-api-access-q72qb" pod "9b8acec7-a35b-41a2-b201-8d2a37fbb439" (UID: "9b8acec7-a35b-41a2-b201-8d2a37fbb439"). InnerVolumeSpecName "kube-api-access-q72qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:18.415992 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.415907    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b8acec7-a35b-41a2-b201-8d2a37fbb439-proxy-tls" pod "9b8acec7-a35b-41a2-b201-8d2a37fbb439" (UID: "9b8acec7-a35b-41a2-b201-8d2a37fbb439"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:18.514222 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.514196    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q72qb\" (UniqueName: \"kubernetes.io/projected/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kube-api-access-q72qb\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:18.514222 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.514222    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b8acec7-a35b-41a2-b201-8d2a37fbb439-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:18.514428 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.514233    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b8acec7-a35b-41a2-b201-8d2a37fbb439-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:18.514428 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.514242    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:18.514428 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.514250    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b8acec7-a35b-41a2-b201-8d2a37fbb439-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:18.844070 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.844035    4947 generic.go:356] "Generic (PLEG): container finished" podID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerID="90d3ff15f261cb4f573638df8da532e573a1d1bab77d8e0fd048664c7dbf358c" exitCode=0
Oct 30 22:53:18.844240 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.844093    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"
Oct 30 22:53:18.844240 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.844123    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" event={"ID":"9b8acec7-a35b-41a2-b201-8d2a37fbb439","Type":"ContainerDied","Data":"90d3ff15f261cb4f573638df8da532e573a1d1bab77d8e0fd048664c7dbf358c"}
Oct 30 22:53:18.844240 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.844176    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972" event={"ID":"9b8acec7-a35b-41a2-b201-8d2a37fbb439","Type":"ContainerDied","Data":"4f9ae05f6582a3fed480e6f16508bee199a1ea47b80dd6f55e707cc7a863fea0"}
Oct 30 22:53:18.844240 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.844201    4947 scope.go:122] "RemoveContainer" containerID="92b5a55b9a7bf274164d15b0a790e63a313826fffd6feb53884a8492ed43ea95"
Oct 30 22:53:18.852794 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.852778    4947 scope.go:122] "RemoveContainer" containerID="90d3ff15f261cb4f573638df8da532e573a1d1bab77d8e0fd048664c7dbf358c"
Oct 30 22:53:18.860137 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.860119    4947 scope.go:122] "RemoveContainer" containerID="10a31539dfa6d0b8e3ae1bc2034f613de791c5c802de3357204bfbcc724c9831"
Oct 30 22:53:18.869487 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.869455    4947 scope.go:122] "RemoveContainer" containerID="92b5a55b9a7bf274164d15b0a790e63a313826fffd6feb53884a8492ed43ea95"
Oct 30 22:53:18.869786 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:53:18.869767    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b5a55b9a7bf274164d15b0a790e63a313826fffd6feb53884a8492ed43ea95\": container with ID starting with 92b5a55b9a7bf274164d15b0a790e63a313826fffd6feb53884a8492ed43ea95 not found: ID does not exist" containerID="92b5a55b9a7bf274164d15b0a790e63a313826fffd6feb53884a8492ed43ea95"
Oct 30 22:53:18.869852 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.869795    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b5a55b9a7bf274164d15b0a790e63a313826fffd6feb53884a8492ed43ea95"} err="failed to get container status \"92b5a55b9a7bf274164d15b0a790e63a313826fffd6feb53884a8492ed43ea95\": rpc error: code = NotFound desc = could not find container \"92b5a55b9a7bf274164d15b0a790e63a313826fffd6feb53884a8492ed43ea95\": container with ID starting with 92b5a55b9a7bf274164d15b0a790e63a313826fffd6feb53884a8492ed43ea95 not found: ID does not exist"
Oct 30 22:53:18.869852 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.869812    4947 scope.go:122] "RemoveContainer" containerID="90d3ff15f261cb4f573638df8da532e573a1d1bab77d8e0fd048664c7dbf358c"
Oct 30 22:53:18.870046 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:53:18.870028    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d3ff15f261cb4f573638df8da532e573a1d1bab77d8e0fd048664c7dbf358c\": container with ID starting with 90d3ff15f261cb4f573638df8da532e573a1d1bab77d8e0fd048664c7dbf358c not found: ID does not exist" containerID="90d3ff15f261cb4f573638df8da532e573a1d1bab77d8e0fd048664c7dbf358c"
Oct 30 22:53:18.870111 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.870056    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d3ff15f261cb4f573638df8da532e573a1d1bab77d8e0fd048664c7dbf358c"} err="failed to get container status \"90d3ff15f261cb4f573638df8da532e573a1d1bab77d8e0fd048664c7dbf358c\": rpc error: code = NotFound desc = could not find container \"90d3ff15f261cb4f573638df8da532e573a1d1bab77d8e0fd048664c7dbf358c\": container with ID starting with 90d3ff15f261cb4f573638df8da532e573a1d1bab77d8e0fd048664c7dbf358c not found: ID does not exist"
Oct 30 22:53:18.870111 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.870078    4947 scope.go:122] "RemoveContainer" containerID="10a31539dfa6d0b8e3ae1bc2034f613de791c5c802de3357204bfbcc724c9831"
Oct 30 22:53:18.870369 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:53:18.870350    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10a31539dfa6d0b8e3ae1bc2034f613de791c5c802de3357204bfbcc724c9831\": container with ID starting with 10a31539dfa6d0b8e3ae1bc2034f613de791c5c802de3357204bfbcc724c9831 not found: ID does not exist" containerID="10a31539dfa6d0b8e3ae1bc2034f613de791c5c802de3357204bfbcc724c9831"
Oct 30 22:53:18.870414 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:18.870374    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a31539dfa6d0b8e3ae1bc2034f613de791c5c802de3357204bfbcc724c9831"} err="failed to get container status \"10a31539dfa6d0b8e3ae1bc2034f613de791c5c802de3357204bfbcc724c9831\": rpc error: code = NotFound desc = could not find container \"10a31539dfa6d0b8e3ae1bc2034f613de791c5c802de3357204bfbcc724c9831\": container with ID starting with 10a31539dfa6d0b8e3ae1bc2034f613de791c5c802de3357204bfbcc724c9831 not found: ID does not exist"
Oct 30 22:53:19.011126 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:19.011094    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"]
Oct 30 22:53:19.060982 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:19.060951    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-promote-predictor-66d68c5b56-tn972"]
Oct 30 22:53:20.158519 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.158489    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" path="/var/lib/kubelet/pods/9b8acec7-a35b-41a2-b201-8d2a37fbb439/volumes"
Oct 30 22:53:20.803631 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.803595    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"]
Oct 30 22:53:20.803928 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.803916    4947 cpu_manager.go:414] "containerMap: removing container" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kserve-container"
Oct 30 22:53:20.803981 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.803930    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kserve-container"
Oct 30 22:53:20.803981 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.803940    4947 cpu_manager.go:414] "containerMap: removing container" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kserve-container"
Oct 30 22:53:20.803981 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.803945    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kserve-container"
Oct 30 22:53:20.803981 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.803955    4947 cpu_manager.go:414] "containerMap: removing container" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="storage-initializer"
Oct 30 22:53:20.803981 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.803960    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="storage-initializer"
Oct 30 22:53:20.803981 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.803968    4947 cpu_manager.go:414] "containerMap: removing container" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kube-rbac-proxy"
Oct 30 22:53:20.803981 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.803972    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kube-rbac-proxy"
Oct 30 22:53:20.803981 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.803981    4947 cpu_manager.go:414] "containerMap: removing container" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kube-rbac-proxy"
Oct 30 22:53:20.804209 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.803986    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kube-rbac-proxy"
Oct 30 22:53:20.804209 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.803993    4947 cpu_manager.go:414] "containerMap: removing container" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="storage-initializer"
Oct 30 22:53:20.804209 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.803998    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="storage-initializer"
Oct 30 22:53:20.804209 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.804053    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kserve-container"
Oct 30 22:53:20.804209 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.804060    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kserve-container"
Oct 30 22:53:20.804209 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.804066    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e80d60-a2d2-4ef9-ba55-8b2bfc44da96" containerName="kube-rbac-proxy"
Oct 30 22:53:20.804209 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.804072    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8acec7-a35b-41a2-b201-8d2a37fbb439" containerName="kube-rbac-proxy"
Oct 30 22:53:20.807171 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.807154    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:20.860784 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.860752    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:53:20.910087 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.910057    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 22:53:20.910087 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.910072    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:53:20.910265 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.910112    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:53:20.910265 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.910136    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:53:20.910265 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.910153    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:53:20.910265 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.910170    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Oct 30 22:53:20.911773 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.911756    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"]
Oct 30 22:53:20.935522 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.935499    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4279c270-03cc-4d1e-bbe1-94711ff1cb60-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:20.935619 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.935530    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:20.935662 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.935625    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:20.935662 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.935653    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4279c270-03cc-4d1e-bbe1-94711ff1cb60-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:20.935755 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:20.935736    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdpdg\" (UniqueName: \"kubernetes.io/projected/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kube-api-access-gdpdg\") pod \"isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:21.036251 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:21.036222    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:21.036367 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:21.036255    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4279c270-03cc-4d1e-bbe1-94711ff1cb60-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:21.036367 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:21.036282    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdpdg\" (UniqueName: \"kubernetes.io/projected/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kube-api-access-gdpdg\") pod \"isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:21.036367 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:21.036313    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4279c270-03cc-4d1e-bbe1-94711ff1cb60-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:21.036367 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:21.036359    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:21.036832 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:21.036808    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:21.036984 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:21.036967    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:21.037060 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:21.037044    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4279c270-03cc-4d1e-bbe1-94711ff1cb60-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:21.038900 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:21.038880    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4279c270-03cc-4d1e-bbe1-94711ff1cb60-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:21.099075 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:21.099008    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdpdg\" (UniqueName: \"kubernetes.io/projected/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kube-api-access-gdpdg\") pod \"isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:21.123872 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:21.123847    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:21.554469 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:21.554437    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"]
Oct 30 22:53:21.557675 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:53:21.557643    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4279c270_03cc_4d1e_bbe1_94711ff1cb60.slice/crio-8adff945406f5beafd6734de17950dcf2b93ca6569b8df5a551f59ff478c8fd1 WatchSource:0}: Error finding container 8adff945406f5beafd6734de17950dcf2b93ca6569b8df5a551f59ff478c8fd1: Status 404 returned error can't find the container with id 8adff945406f5beafd6734de17950dcf2b93ca6569b8df5a551f59ff478c8fd1
Oct 30 22:53:21.858991 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:21.858911    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6" event={"ID":"4279c270-03cc-4d1e-bbe1-94711ff1cb60","Type":"ContainerStarted","Data":"30ec07c9135266ee15c8c95efac97a28ddf5a70d376564b9706aa814ae59ab42"}
Oct 30 22:53:21.858991 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:21.858969    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6" event={"ID":"4279c270-03cc-4d1e-bbe1-94711ff1cb60","Type":"ContainerStarted","Data":"8adff945406f5beafd6734de17950dcf2b93ca6569b8df5a551f59ff478c8fd1"}
Oct 30 22:53:22.390613 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:22.390587    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:53:22.396618 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:22.396591    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:53:25.876050 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:25.875963    4947 generic.go:356] "Generic (PLEG): container finished" podID="4279c270-03cc-4d1e-bbe1-94711ff1cb60" containerID="30ec07c9135266ee15c8c95efac97a28ddf5a70d376564b9706aa814ae59ab42" exitCode=0
Oct 30 22:53:25.876477 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:25.876040    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6" event={"ID":"4279c270-03cc-4d1e-bbe1-94711ff1cb60","Type":"ContainerDied","Data":"30ec07c9135266ee15c8c95efac97a28ddf5a70d376564b9706aa814ae59ab42"}
Oct 30 22:53:26.882262 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:26.882225    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6" event={"ID":"4279c270-03cc-4d1e-bbe1-94711ff1cb60","Type":"ContainerStarted","Data":"a90a0f6fde263641ce362bdb547ea2fc637b6207ba7a050e386928e4515366f1"}
Oct 30 22:53:26.882671 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:26.882293    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6" event={"ID":"4279c270-03cc-4d1e-bbe1-94711ff1cb60","Type":"ContainerStarted","Data":"e36c09b90d191a9ec59e743ddafa3b0204763d7804d55fc901618b34ce179c09"}
Oct 30 22:53:26.882671 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:26.882516    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:27.004789 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:27.004735    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6" podStartSLOduration=3.011330057 podStartE2EDuration="7.004718568s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.993388511s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:27.004356709 +0000 UTC m=+2705.003704270" watchObservedRunningTime="2026-10-30 22:53:27.004718568 +0000 UTC m=+2705.004066131"
Oct 30 22:53:27.887299 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:27.887262    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:29.048149 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:53:29.048110    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:53:33.897000 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:53:33.896969    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:53:59.047346 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:53:59.047299    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:54:03.900774 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:03.900670    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6" podUID="4279c270-03cc-4d1e-bbe1-94711ff1cb60" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 30 22:54:13.900299 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:13.900271    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:54:20.892829 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:20.892784    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"]
Oct 30 22:54:20.893357 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:20.893058    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6" podUID="4279c270-03cc-4d1e-bbe1-94711ff1cb60" containerName="kserve-container" containerID="cri-o://e36c09b90d191a9ec59e743ddafa3b0204763d7804d55fc901618b34ce179c09" gracePeriod=30
Oct 30 22:54:20.893357 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:20.893165    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6" podUID="4279c270-03cc-4d1e-bbe1-94711ff1cb60" containerName="kube-rbac-proxy" containerID="cri-o://a90a0f6fde263641ce362bdb547ea2fc637b6207ba7a050e386928e4515366f1" gracePeriod=30
Oct 30 22:54:21.109500 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:21.109463    4947 generic.go:356] "Generic (PLEG): container finished" podID="4279c270-03cc-4d1e-bbe1-94711ff1cb60" containerID="a90a0f6fde263641ce362bdb547ea2fc637b6207ba7a050e386928e4515366f1" exitCode=0
Oct 30 22:54:21.109668 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:21.109529    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6" event={"ID":"4279c270-03cc-4d1e-bbe1-94711ff1cb60","Type":"ContainerDied","Data":"a90a0f6fde263641ce362bdb547ea2fc637b6207ba7a050e386928e4515366f1"}
Oct 30 22:54:23.892110 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:23.892073    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6" podUID="4279c270-03cc-4d1e-bbe1-94711ff1cb60" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.64:8643/healthz\": dial tcp 10.132.0.64:8643: connect: connection refused"
Oct 30 22:54:28.144367 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.144320    4947 generic.go:356] "Generic (PLEG): container finished" podID="4279c270-03cc-4d1e-bbe1-94711ff1cb60" containerID="e36c09b90d191a9ec59e743ddafa3b0204763d7804d55fc901618b34ce179c09" exitCode=0
Oct 30 22:54:28.144796 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.144363    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6" event={"ID":"4279c270-03cc-4d1e-bbe1-94711ff1cb60","Type":"ContainerDied","Data":"e36c09b90d191a9ec59e743ddafa3b0204763d7804d55fc901618b34ce179c09"}
Oct 30 22:54:28.144796 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.144470    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6" event={"ID":"4279c270-03cc-4d1e-bbe1-94711ff1cb60","Type":"ContainerDied","Data":"8adff945406f5beafd6734de17950dcf2b93ca6569b8df5a551f59ff478c8fd1"}
Oct 30 22:54:28.144796 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.144503    4947 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8adff945406f5beafd6734de17950dcf2b93ca6569b8df5a551f59ff478c8fd1"
Oct 30 22:54:28.157046 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.157022    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:54:28.247889 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.247812    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4279c270-03cc-4d1e-bbe1-94711ff1cb60-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4279c270-03cc-4d1e-bbe1-94711ff1cb60-cabundle-cert\") pod \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") "
Oct 30 22:54:28.247889 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.247851    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kserve-provision-location\") pod \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") "
Oct 30 22:54:28.248115 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.247920    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4279c270-03cc-4d1e-bbe1-94711ff1cb60-proxy-tls\" (UniqueName: \"kubernetes.io/secret/4279c270-03cc-4d1e-bbe1-94711ff1cb60-proxy-tls\") pod \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") "
Oct 30 22:54:28.248115 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.247946    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kube-api-access-gdpdg\" (UniqueName: \"kubernetes.io/projected/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kube-api-access-gdpdg\") pod \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") "
Oct 30 22:54:28.248115 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.247993    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kube-rbac-proxy-sar-config\") pod \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\" (UID: \"4279c270-03cc-4d1e-bbe1-94711ff1cb60\") "
Oct 30 22:54:28.248280 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.248164    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4279c270-03cc-4d1e-bbe1-94711ff1cb60-cabundle-cert" pod "4279c270-03cc-4d1e-bbe1-94711ff1cb60" (UID: "4279c270-03cc-4d1e-bbe1-94711ff1cb60"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:28.248280 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.248189    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kserve-provision-location" pod "4279c270-03cc-4d1e-bbe1-94711ff1cb60" (UID: "4279c270-03cc-4d1e-bbe1-94711ff1cb60"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:28.248554 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.248528    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kube-rbac-proxy-sar-config" pod "4279c270-03cc-4d1e-bbe1-94711ff1cb60" (UID: "4279c270-03cc-4d1e-bbe1-94711ff1cb60"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:28.250215 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.250187    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kube-api-access-gdpdg" pod "4279c270-03cc-4d1e-bbe1-94711ff1cb60" (UID: "4279c270-03cc-4d1e-bbe1-94711ff1cb60"). InnerVolumeSpecName "kube-api-access-gdpdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:28.250372 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.250309    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4279c270-03cc-4d1e-bbe1-94711ff1cb60-proxy-tls" pod "4279c270-03cc-4d1e-bbe1-94711ff1cb60" (UID: "4279c270-03cc-4d1e-bbe1-94711ff1cb60"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:28.349091 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.349055    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4279c270-03cc-4d1e-bbe1-94711ff1cb60-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:28.349238 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.349108    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdpdg\" (UniqueName: \"kubernetes.io/projected/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kube-api-access-gdpdg\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:28.349238 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.349125    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:28.349238 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.349140    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4279c270-03cc-4d1e-bbe1-94711ff1cb60-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:28.349238 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:28.349153    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4279c270-03cc-4d1e-bbe1-94711ff1cb60-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:29.048649 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:54:29.048612    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:54:29.148703 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:29.148669    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"
Oct 30 22:54:29.366184 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:29.366105    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"]
Oct 30 22:54:29.415809 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:29.415771    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-runtime-predictor-7c447f58fb-5zzd6"]
Oct 30 22:54:30.158218 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:30.158186    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4279c270-03cc-4d1e-bbe1-94711ff1cb60" path="/var/lib/kubelet/pods/4279c270-03cc-4d1e-bbe1-94711ff1cb60/volumes"
Oct 30 22:54:32.064348 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.064297    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"]
Oct 30 22:54:32.064762 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.064660    4947 cpu_manager.go:414] "containerMap: removing container" podUID="4279c270-03cc-4d1e-bbe1-94711ff1cb60" containerName="kserve-container"
Oct 30 22:54:32.064762 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.064671    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4279c270-03cc-4d1e-bbe1-94711ff1cb60" containerName="kserve-container"
Oct 30 22:54:32.064762 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.064691    4947 cpu_manager.go:414] "containerMap: removing container" podUID="4279c270-03cc-4d1e-bbe1-94711ff1cb60" containerName="kube-rbac-proxy"
Oct 30 22:54:32.064762 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.064696    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4279c270-03cc-4d1e-bbe1-94711ff1cb60" containerName="kube-rbac-proxy"
Oct 30 22:54:32.064762 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.064720    4947 cpu_manager.go:414] "containerMap: removing container" podUID="4279c270-03cc-4d1e-bbe1-94711ff1cb60" containerName="storage-initializer"
Oct 30 22:54:32.064762 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.064726    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4279c270-03cc-4d1e-bbe1-94711ff1cb60" containerName="storage-initializer"
Oct 30 22:54:32.064973 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.064794    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="4279c270-03cc-4d1e-bbe1-94711ff1cb60" containerName="kserve-container"
Oct 30 22:54:32.064973 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.064803    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="4279c270-03cc-4d1e-bbe1-94711ff1cb60" containerName="kube-rbac-proxy"
Oct 30 22:54:32.069247 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.069223    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:32.161450 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.161418    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Oct 30 22:54:32.161450 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.161449    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:54:32.161633 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.161469    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:54:32.161633 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.161471    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:54:32.161633 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.161506    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 22:54:32.172232 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.172215    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Oct 30 22:54:32.172309 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.172261    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:54:32.173722 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.173705    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"]
Oct 30 22:54:32.180731 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.180713    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3de663c-a635-40f4-a536-ad32fcb3473a-proxy-tls\") pod \"isvc-xgboost-v2-predictor-7cfc9c5487-l69jx\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:32.180825 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.180743    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3de663c-a635-40f4-a536-ad32fcb3473a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-7cfc9c5487-l69jx\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:32.180825 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.180765    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3de663c-a635-40f4-a536-ad32fcb3473a-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-7cfc9c5487-l69jx\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:32.180954 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.180911    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3de663c-a635-40f4-a536-ad32fcb3473a-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-7cfc9c5487-l69jx\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:32.180954 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.180936    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8dp8\" (UniqueName: \"kubernetes.io/projected/e3de663c-a635-40f4-a536-ad32fcb3473a-kube-api-access-v8dp8\") pod \"isvc-xgboost-v2-predictor-7cfc9c5487-l69jx\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:32.282256 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.282225    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3de663c-a635-40f4-a536-ad32fcb3473a-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-7cfc9c5487-l69jx\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:32.282256 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.282255    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8dp8\" (UniqueName: \"kubernetes.io/projected/e3de663c-a635-40f4-a536-ad32fcb3473a-kube-api-access-v8dp8\") pod \"isvc-xgboost-v2-predictor-7cfc9c5487-l69jx\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:32.282455 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.282290    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3de663c-a635-40f4-a536-ad32fcb3473a-proxy-tls\") pod \"isvc-xgboost-v2-predictor-7cfc9c5487-l69jx\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:32.282455 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.282383    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3de663c-a635-40f4-a536-ad32fcb3473a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-7cfc9c5487-l69jx\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:32.282455 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.282422    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3de663c-a635-40f4-a536-ad32fcb3473a-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-7cfc9c5487-l69jx\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:32.282820 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.282793    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3de663c-a635-40f4-a536-ad32fcb3473a-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-7cfc9c5487-l69jx\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:32.283032 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.283016    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3de663c-a635-40f4-a536-ad32fcb3473a-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-7cfc9c5487-l69jx\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:32.283081 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.283018    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3de663c-a635-40f4-a536-ad32fcb3473a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-7cfc9c5487-l69jx\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:32.284861 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.284838    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3de663c-a635-40f4-a536-ad32fcb3473a-proxy-tls\") pod \"isvc-xgboost-v2-predictor-7cfc9c5487-l69jx\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:32.344729 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.344663    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8dp8\" (UniqueName: \"kubernetes.io/projected/e3de663c-a635-40f4-a536-ad32fcb3473a-kube-api-access-v8dp8\") pod \"isvc-xgboost-v2-predictor-7cfc9c5487-l69jx\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") " pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:32.386066 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.386037    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:32.828714 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:32.828681    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"]
Oct 30 22:54:32.831563 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:54:32.831538    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3de663c_a635_40f4_a536_ad32fcb3473a.slice/crio-aeda8d2df8939d45ae9893d385cda85663944882d508ba8737cadc3a071c50cd WatchSource:0}: Error finding container aeda8d2df8939d45ae9893d385cda85663944882d508ba8737cadc3a071c50cd: Status 404 returned error can't find the container with id aeda8d2df8939d45ae9893d385cda85663944882d508ba8737cadc3a071c50cd
Oct 30 22:54:33.166366 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:33.166263    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" event={"ID":"e3de663c-a635-40f4-a536-ad32fcb3473a","Type":"ContainerStarted","Data":"beaf536ad1e3f9b58f8e233fbbce8bbf91983841dcc8f86c0847da419114632e"}
Oct 30 22:54:33.166366 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:33.166315    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" event={"ID":"e3de663c-a635-40f4-a536-ad32fcb3473a","Type":"ContainerStarted","Data":"aeda8d2df8939d45ae9893d385cda85663944882d508ba8737cadc3a071c50cd"}
Oct 30 22:54:37.183125 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:37.183079    4947 generic.go:356] "Generic (PLEG): container finished" podID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerID="beaf536ad1e3f9b58f8e233fbbce8bbf91983841dcc8f86c0847da419114632e" exitCode=0
Oct 30 22:54:37.183547 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:37.183151    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" event={"ID":"e3de663c-a635-40f4-a536-ad32fcb3473a","Type":"ContainerDied","Data":"beaf536ad1e3f9b58f8e233fbbce8bbf91983841dcc8f86c0847da419114632e"}
Oct 30 22:54:38.188361 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:38.188313    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" event={"ID":"e3de663c-a635-40f4-a536-ad32fcb3473a","Type":"ContainerStarted","Data":"44f94044a1f2c322eeafea5ee1c148e7d7b365cf3ceb0a564a1ad7c4c9818667"}
Oct 30 22:54:38.188827 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:38.188382    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" event={"ID":"e3de663c-a635-40f4-a536-ad32fcb3473a","Type":"ContainerStarted","Data":"01f297cc82e513c1efd40d9b2c3bb9a307488245e908deed43e427626a8d093f"}
Oct 30 22:54:38.188827 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:38.188711    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:38.188911 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:38.188842    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:38.190343 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:38.190305    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 22:54:38.310028 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:38.309964    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" podStartSLOduration=2.433899457 podStartE2EDuration="6.30994838s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.876048923s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:54:38.308827774 +0000 UTC m=+2776.308175334" watchObservedRunningTime="2026-10-30 22:54:38.30994838 +0000 UTC m=+2776.309296006"
Oct 30 22:54:39.192369 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:39.192311    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 22:54:40.196536 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:40.196486    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 22:54:44.432218 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:44.432185    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"]
Oct 30 22:54:44.432741 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:44.432436    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kserve-container" containerID="cri-o://67409ee9207c4007ed4ab8d4d1651aefa2071a842a7890bbf005f41c9bbcb777" gracePeriod=30
Oct 30 22:54:44.432741 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:44.432463    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kube-rbac-proxy" containerID="cri-o://88b7c2d5e0a32bbe315209d0b82e497f417a372610d09697c361097d89fea532" gracePeriod=30
Oct 30 22:54:45.201283 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:45.201249    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:54:45.201988 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:45.201958    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 22:54:45.221082 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:45.221054    4947 generic.go:356] "Generic (PLEG): container finished" podID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerID="88b7c2d5e0a32bbe315209d0b82e497f417a372610d09697c361097d89fea532" exitCode=0
Oct 30 22:54:45.221215 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:45.221088    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" event={"ID":"6fdc4ce5-a233-4a9c-9767-f771db1cb82c","Type":"ContainerDied","Data":"88b7c2d5e0a32bbe315209d0b82e497f417a372610d09697c361097d89fea532"}
Oct 30 22:54:46.448194 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:46.448161    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.61:8643/healthz\": dial tcp 10.132.0.61:8643: connect: connection refused"
Oct 30 22:54:49.241235 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.241205    4947 generic.go:356] "Generic (PLEG): container finished" podID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerID="67409ee9207c4007ed4ab8d4d1651aefa2071a842a7890bbf005f41c9bbcb777" exitCode=0
Oct 30 22:54:49.241608 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.241281    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" event={"ID":"6fdc4ce5-a233-4a9c-9767-f771db1cb82c","Type":"ContainerDied","Data":"67409ee9207c4007ed4ab8d4d1651aefa2071a842a7890bbf005f41c9bbcb777"}
Oct 30 22:54:49.283987 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.283965    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:54:49.432251 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.432171    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kserve-provision-location\") pod \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") "
Oct 30 22:54:49.432251 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.432223    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kube-api-access-c79qq\" (UniqueName: \"kubernetes.io/projected/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kube-api-access-c79qq\") pod \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") "
Oct 30 22:54:49.432251 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.432245    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-cabundle-cert\") pod \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") "
Oct 30 22:54:49.432577 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.432273    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-proxy-tls\") pod \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") "
Oct 30 22:54:49.432577 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.432369    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kube-rbac-proxy-sar-config\") pod \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\" (UID: \"6fdc4ce5-a233-4a9c-9767-f771db1cb82c\") "
Oct 30 22:54:49.432577 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.432549    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kserve-provision-location" pod "6fdc4ce5-a233-4a9c-9767-f771db1cb82c" (UID: "6fdc4ce5-a233-4a9c-9767-f771db1cb82c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:49.432765 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.432742    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-cabundle-cert" pod "6fdc4ce5-a233-4a9c-9767-f771db1cb82c" (UID: "6fdc4ce5-a233-4a9c-9767-f771db1cb82c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:49.432882 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.432853    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kube-rbac-proxy-sar-config" pod "6fdc4ce5-a233-4a9c-9767-f771db1cb82c" (UID: "6fdc4ce5-a233-4a9c-9767-f771db1cb82c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:49.434427 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.434409    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kube-api-access-c79qq" pod "6fdc4ce5-a233-4a9c-9767-f771db1cb82c" (UID: "6fdc4ce5-a233-4a9c-9767-f771db1cb82c"). InnerVolumeSpecName "kube-api-access-c79qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:49.434513 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.434494    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-proxy-tls" pod "6fdc4ce5-a233-4a9c-9767-f771db1cb82c" (UID: "6fdc4ce5-a233-4a9c-9767-f771db1cb82c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:49.533289 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.533250    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c79qq\" (UniqueName: \"kubernetes.io/projected/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kube-api-access-c79qq\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:49.533289 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.533278    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:49.533289 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.533289    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:49.533523 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.533298    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:49.533523 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:49.533307    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fdc4ce5-a233-4a9c-9767-f771db1cb82c-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:50.247819 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:50.247789    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g" event={"ID":"6fdc4ce5-a233-4a9c-9767-f771db1cb82c","Type":"ContainerDied","Data":"8870f39b5e0f03d9d64797cce34f4ba52ce651e4686f058eac2ea89491121b74"}
Oct 30 22:54:50.248214 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:50.247854    4947 scope.go:122] "RemoveContainer" containerID="88b7c2d5e0a32bbe315209d0b82e497f417a372610d09697c361097d89fea532"
Oct 30 22:54:50.248214 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:50.247799    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"
Oct 30 22:54:50.255953 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:50.255932    4947 scope.go:122] "RemoveContainer" containerID="67409ee9207c4007ed4ab8d4d1651aefa2071a842a7890bbf005f41c9bbcb777"
Oct 30 22:54:50.263219 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:50.263194    4947 scope.go:122] "RemoveContainer" containerID="a9ed98562ff9b2e4c96b5fd5e245cb2a5487a3bf6714bba02d6d4368c862e875"
Oct 30 22:54:50.465685 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:50.465598    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"]
Oct 30 22:54:50.515251 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:50.515216    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-promote-v2-predictor-66ffb95d77-2s76g"]
Oct 30 22:54:52.157957 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:52.157927    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" path="/var/lib/kubelet/pods/6fdc4ce5-a233-4a9c-9767-f771db1cb82c/volumes"
Oct 30 22:54:53.573897 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.573866    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"]
Oct 30 22:54:53.574270 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.574180    4947 cpu_manager.go:414] "containerMap: removing container" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kserve-container"
Oct 30 22:54:53.574270 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.574190    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kserve-container"
Oct 30 22:54:53.574270 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.574208    4947 cpu_manager.go:414] "containerMap: removing container" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="storage-initializer"
Oct 30 22:54:53.574270 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.574212    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="storage-initializer"
Oct 30 22:54:53.574270 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.574230    4947 cpu_manager.go:414] "containerMap: removing container" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kube-rbac-proxy"
Oct 30 22:54:53.574270 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.574234    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kube-rbac-proxy"
Oct 30 22:54:53.574477 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.574290    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kube-rbac-proxy"
Oct 30 22:54:53.574477 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.574299    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fdc4ce5-a233-4a9c-9767-f771db1cb82c" containerName="kserve-container"
Oct 30 22:54:53.577475 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.577458    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:53.631764 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.631724    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"seaweedfs-s3-creds\""
Oct 30 22:54:53.631966 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.631787    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"default-dockercfg-s9cqw\""
Oct 30 22:54:53.667593 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.667557    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85gtc\" (UniqueName: \"kubernetes.io/projected/93a9abf2-7c97-4999-960e-2005cdf94f34-kube-api-access-85gtc\") pod \"isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:53.667748 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.667609    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a9abf2-7c97-4999-960e-2005cdf94f34-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:53.667748 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.667637    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a9abf2-7c97-4999-960e-2005cdf94f34-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:53.667748 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.667715    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a9abf2-7c97-4999-960e-2005cdf94f34-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:53.667857 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.667821    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a9abf2-7c97-4999-960e-2005cdf94f34-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:53.680815 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.680784    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Oct 30 22:54:53.680815 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.680793    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"kube-root-ca.crt\""
Oct 30 22:54:53.680990 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.680823    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"openshift-service-ca.crt\""
Oct 30 22:54:53.680990 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.680842    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Oct 30 22:54:53.680990 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.680865    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:54:53.682300 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.682281    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"]
Oct 30 22:54:53.768588 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.768554    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a9abf2-7c97-4999-960e-2005cdf94f34-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:53.768768 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.768597    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a9abf2-7c97-4999-960e-2005cdf94f34-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:53.768768 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.768732    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85gtc\" (UniqueName: \"kubernetes.io/projected/93a9abf2-7c97-4999-960e-2005cdf94f34-kube-api-access-85gtc\") pod \"isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:53.768895 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.768806    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a9abf2-7c97-4999-960e-2005cdf94f34-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:53.768895 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.768853    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a9abf2-7c97-4999-960e-2005cdf94f34-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:53.769001 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.768926    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a9abf2-7c97-4999-960e-2005cdf94f34-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:53.769311 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.769292    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a9abf2-7c97-4999-960e-2005cdf94f34-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:53.769429 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.769384    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a9abf2-7c97-4999-960e-2005cdf94f34-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:53.771395 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.771374    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a9abf2-7c97-4999-960e-2005cdf94f34-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:53.831778 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.831715    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85gtc\" (UniqueName: \"kubernetes.io/projected/93a9abf2-7c97-4999-960e-2005cdf94f34-kube-api-access-85gtc\") pod \"isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:53.893452 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:53.893423    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:54.321804 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.321750    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"]
Oct 30 22:54:54.327405 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:54:54.327380    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a9abf2_7c97_4999_960e_2005cdf94f34.slice/crio-525f3f3075819e932180773230f9f123599b3c6208f652c13c9c9cd03fe2b7b5 WatchSource:0}: Error finding container 525f3f3075819e932180773230f9f123599b3c6208f652c13c9c9cd03fe2b7b5: Status 404 returned error can't find the container with id 525f3f3075819e932180773230f9f123599b3c6208f652c13c9c9cd03fe2b7b5
Oct 30 22:54:54.555631 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.555598    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"]
Oct 30 22:54:54.559093 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.559078    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:54:54.613152 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.613070    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-canary-rollback-predictor-serving-cert\""
Oct 30 22:54:54.621400 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.621370    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"]
Oct 30 22:54:54.675220 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.675190    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5597787b4-4vghx\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:54:54.675437 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.675236    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5597787b4-4vghx\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:54:54.675437 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.675258    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5ddbfd1-213f-44ee-9b05-ec6477bac135-proxy-tls\") pod \"isvc-canary-rollback-predictor-5597787b4-4vghx\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:54:54.675437 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.675405    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p7jz\" (UniqueName: \"kubernetes.io/projected/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kube-api-access-5p7jz\") pod \"isvc-canary-rollback-predictor-5597787b4-4vghx\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:54:54.675437 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.675429    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5ddbfd1-213f-44ee-9b05-ec6477bac135-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5597787b4-4vghx\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:54:54.776177 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.776140    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5597787b4-4vghx\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:54:54.776412 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.776210    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5597787b4-4vghx\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:54:54.776412 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.776244    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5ddbfd1-213f-44ee-9b05-ec6477bac135-proxy-tls\") pod \"isvc-canary-rollback-predictor-5597787b4-4vghx\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:54:54.776412 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.776288    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p7jz\" (UniqueName: \"kubernetes.io/projected/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kube-api-access-5p7jz\") pod \"isvc-canary-rollback-predictor-5597787b4-4vghx\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:54:54.776412 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.776319    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5ddbfd1-213f-44ee-9b05-ec6477bac135-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5597787b4-4vghx\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:54:54.776797 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.776761    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5597787b4-4vghx\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:54:54.777031 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.777009    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5597787b4-4vghx\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:54:54.777279 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.777256    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5ddbfd1-213f-44ee-9b05-ec6477bac135-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5597787b4-4vghx\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:54:54.779326 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.779306    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5ddbfd1-213f-44ee-9b05-ec6477bac135-proxy-tls\") pod \"isvc-canary-rollback-predictor-5597787b4-4vghx\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:54:54.839885 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.839851    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p7jz\" (UniqueName: \"kubernetes.io/projected/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kube-api-access-5p7jz\") pod \"isvc-canary-rollback-predictor-5597787b4-4vghx\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") " pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:54:54.876635 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:54.876551    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:54:55.202238 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:55.202163    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 22:54:55.269879 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:55.269847    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" event={"ID":"93a9abf2-7c97-4999-960e-2005cdf94f34","Type":"ContainerStarted","Data":"95cd5691924eb8cd1a734358290dadb32c2c0d4e981acb7d276ee3eb35d013eb"}
Oct 30 22:54:55.270031 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:55.269906    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" event={"ID":"93a9abf2-7c97-4999-960e-2005cdf94f34","Type":"ContainerStarted","Data":"525f3f3075819e932180773230f9f123599b3c6208f652c13c9c9cd03fe2b7b5"}
Oct 30 22:54:55.305938 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:55.305899    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"]
Oct 30 22:54:55.308230 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:54:55.308204    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ddbfd1_213f_44ee_9b05_ec6477bac135.slice/crio-6c8c094f25a7833af0525a2ce536a15518b3f5ff46fb32017bea4615b17d7244 WatchSource:0}: Error finding container 6c8c094f25a7833af0525a2ce536a15518b3f5ff46fb32017bea4615b17d7244: Status 404 returned error can't find the container with id 6c8c094f25a7833af0525a2ce536a15518b3f5ff46fb32017bea4615b17d7244
Oct 30 22:54:56.275643 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:56.275607    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" event={"ID":"d5ddbfd1-213f-44ee-9b05-ec6477bac135","Type":"ContainerStarted","Data":"234b818c2353f00dc29761676120efe8a13c4e79a67333c8c2277caa006a3570"}
Oct 30 22:54:56.276099 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:56.275668    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" event={"ID":"d5ddbfd1-213f-44ee-9b05-ec6477bac135","Type":"ContainerStarted","Data":"6c8c094f25a7833af0525a2ce536a15518b3f5ff46fb32017bea4615b17d7244"}
Oct 30 22:54:58.285476 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:58.285445    4947 generic.go:356] "Generic (PLEG): container finished" podID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerID="95cd5691924eb8cd1a734358290dadb32c2c0d4e981acb7d276ee3eb35d013eb" exitCode=0
Oct 30 22:54:58.285834 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:58.285497    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" event={"ID":"93a9abf2-7c97-4999-960e-2005cdf94f34","Type":"ContainerDied","Data":"95cd5691924eb8cd1a734358290dadb32c2c0d4e981acb7d276ee3eb35d013eb"}
Oct 30 22:54:59.048734 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:54:59.048700    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:54:59.293296 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:59.293268    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" event={"ID":"93a9abf2-7c97-4999-960e-2005cdf94f34","Type":"ContainerStarted","Data":"17800f07df0df030dbfcec665ec69e31a5fe6285f28ab42b3db061fc3a2634bd"}
Oct 30 22:54:59.293723 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:59.293327    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" event={"ID":"93a9abf2-7c97-4999-960e-2005cdf94f34","Type":"ContainerStarted","Data":"49cd188a37a556c254ae43011f97765d926eb611aa8e75cd5ad4070e94a61285"}
Oct 30 22:54:59.293723 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:59.293676    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:59.293833 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:59.293816    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:54:59.294887 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:59.294867    4947 generic.go:356] "Generic (PLEG): container finished" podID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerID="234b818c2353f00dc29761676120efe8a13c4e79a67333c8c2277caa006a3570" exitCode=0
Oct 30 22:54:59.295000 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:59.294945    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" event={"ID":"d5ddbfd1-213f-44ee-9b05-ec6477bac135","Type":"ContainerDied","Data":"234b818c2353f00dc29761676120efe8a13c4e79a67333c8c2277caa006a3570"}
Oct 30 22:54:59.295579 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:59.295548    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 22:54:59.415235 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:54:59.415126    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" podStartSLOduration=2.724665865 podStartE2EDuration="6.415108436s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.690442571s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:54:59.413775031 +0000 UTC m=+2797.413122596" watchObservedRunningTime="2026-10-30 22:54:59.415108436 +0000 UTC m=+2797.414455999"
Oct 30 22:55:00.309908 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:00.309874    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" event={"ID":"d5ddbfd1-213f-44ee-9b05-ec6477bac135","Type":"ContainerStarted","Data":"f0dd2e1711016def728b684172999320125fd7b8bb3f29bce0d8d1445c064493"}
Oct 30 22:55:00.310354 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:00.309953    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" event={"ID":"d5ddbfd1-213f-44ee-9b05-ec6477bac135","Type":"ContainerStarted","Data":"a8a3f7587bbd8b86fbae8744a3792a1ce550deb78828b81ceb219aa6c2ad59f3"}
Oct 30 22:55:00.310354 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:00.310155    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 22:55:00.310460 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:00.310430    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:55:00.430820 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:00.430770    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" podStartSLOduration=2.695906663 podStartE2EDuration="6.430757063s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.7348504s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:00.429911123 +0000 UTC m=+2798.429258705" watchObservedRunningTime="2026-10-30 22:55:00.430757063 +0000 UTC m=+2798.430104622"
Oct 30 22:55:01.314377 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:01.314351    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:55:01.315643 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:01.315621    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 22:55:02.318137 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:02.318102    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 22:55:05.201834 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:05.201786    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 22:55:05.315144 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:05.315117    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:55:05.315703 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:05.315674    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 22:55:07.322907 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:07.322879    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:55:07.323553 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:07.323523    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 22:55:15.201975 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:15.201930    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 22:55:15.316222 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:15.316182    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 22:55:17.324510 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:17.324469    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 22:55:25.202570 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:25.202526    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 22:55:25.315861 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:25.315826    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 22:55:27.323744 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:27.323707    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 22:55:29.048400 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:55:29.048365    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:55:35.202625 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:35.202524    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 22:55:35.315611 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:35.315570    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 22:55:37.323847 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:37.323806    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 22:55:45.202542 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:45.202509    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:55:45.316090 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:45.316056    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 22:55:47.324404 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:47.324366    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 22:55:52.960944 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:52.960908    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"]
Oct 30 22:55:52.961494 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:52.961141    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kserve-container" containerID="cri-o://01f297cc82e513c1efd40d9b2c3bb9a307488245e908deed43e427626a8d093f" gracePeriod=30
Oct 30 22:55:52.961494 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:52.961194    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kube-rbac-proxy" containerID="cri-o://44f94044a1f2c322eeafea5ee1c148e7d7b365cf3ceb0a564a1ad7c4c9818667" gracePeriod=30
Oct 30 22:55:53.539291 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:53.539251    4947 generic.go:356] "Generic (PLEG): container finished" podID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerID="44f94044a1f2c322eeafea5ee1c148e7d7b365cf3ceb0a564a1ad7c4c9818667" exitCode=0
Oct 30 22:55:53.539480 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:53.539324    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" event={"ID":"e3de663c-a635-40f4-a536-ad32fcb3473a","Type":"ContainerDied","Data":"44f94044a1f2c322eeafea5ee1c148e7d7b365cf3ceb0a564a1ad7c4c9818667"}
Oct 30 22:55:55.196737 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:55.196692    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.65:8643/healthz\": dial tcp 10.132.0.65:8643: connect: connection refused"
Oct 30 22:55:55.202182 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:55.202140    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 22:55:55.315828 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:55.315792    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 22:55:57.323550 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:57.323508    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 22:55:57.814557 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:57.814530    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:55:57.850102 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:57.850077    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3de663c-a635-40f4-a536-ad32fcb3473a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3de663c-a635-40f4-a536-ad32fcb3473a-kube-rbac-proxy-sar-config\") pod \"e3de663c-a635-40f4-a536-ad32fcb3473a\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") "
Oct 30 22:55:57.850239 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:57.850114    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3de663c-a635-40f4-a536-ad32fcb3473a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3de663c-a635-40f4-a536-ad32fcb3473a-cabundle-cert\") pod \"e3de663c-a635-40f4-a536-ad32fcb3473a\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") "
Oct 30 22:55:57.850239 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:57.850164    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3de663c-a635-40f4-a536-ad32fcb3473a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3de663c-a635-40f4-a536-ad32fcb3473a-kserve-provision-location\") pod \"e3de663c-a635-40f4-a536-ad32fcb3473a\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") "
Oct 30 22:55:57.850239 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:57.850210    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3de663c-a635-40f4-a536-ad32fcb3473a-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3de663c-a635-40f4-a536-ad32fcb3473a-proxy-tls\") pod \"e3de663c-a635-40f4-a536-ad32fcb3473a\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") "
Oct 30 22:55:57.850239 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:57.850228    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3de663c-a635-40f4-a536-ad32fcb3473a-kube-api-access-v8dp8\" (UniqueName: \"kubernetes.io/projected/e3de663c-a635-40f4-a536-ad32fcb3473a-kube-api-access-v8dp8\") pod \"e3de663c-a635-40f4-a536-ad32fcb3473a\" (UID: \"e3de663c-a635-40f4-a536-ad32fcb3473a\") "
Oct 30 22:55:57.850564 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:57.850536    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3de663c-a635-40f4-a536-ad32fcb3473a-kube-rbac-proxy-sar-config" pod "e3de663c-a635-40f4-a536-ad32fcb3473a" (UID: "e3de663c-a635-40f4-a536-ad32fcb3473a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:57.850627 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:57.850560    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3de663c-a635-40f4-a536-ad32fcb3473a-kserve-provision-location" pod "e3de663c-a635-40f4-a536-ad32fcb3473a" (UID: "e3de663c-a635-40f4-a536-ad32fcb3473a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:55:57.850627 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:57.850607    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3de663c-a635-40f4-a536-ad32fcb3473a-cabundle-cert" pod "e3de663c-a635-40f4-a536-ad32fcb3473a" (UID: "e3de663c-a635-40f4-a536-ad32fcb3473a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:57.852472 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:57.852450    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3de663c-a635-40f4-a536-ad32fcb3473a-proxy-tls" pod "e3de663c-a635-40f4-a536-ad32fcb3473a" (UID: "e3de663c-a635-40f4-a536-ad32fcb3473a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:57.852472 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:57.852456    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3de663c-a635-40f4-a536-ad32fcb3473a-kube-api-access-v8dp8" pod "e3de663c-a635-40f4-a536-ad32fcb3473a" (UID: "e3de663c-a635-40f4-a536-ad32fcb3473a"). InnerVolumeSpecName "kube-api-access-v8dp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:57.951114 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:57.951082    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3de663c-a635-40f4-a536-ad32fcb3473a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:57.951114 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:57.951112    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3de663c-a635-40f4-a536-ad32fcb3473a-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:57.951291 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:57.951125    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3de663c-a635-40f4-a536-ad32fcb3473a-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:57.951291 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:57.951138    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3de663c-a635-40f4-a536-ad32fcb3473a-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:57.951291 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:57.951153    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8dp8\" (UniqueName: \"kubernetes.io/projected/e3de663c-a635-40f4-a536-ad32fcb3473a-kube-api-access-v8dp8\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:58.562834 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:58.562797    4947 generic.go:356] "Generic (PLEG): container finished" podID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerID="01f297cc82e513c1efd40d9b2c3bb9a307488245e908deed43e427626a8d093f" exitCode=0
Oct 30 22:55:58.563389 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:58.562860    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"
Oct 30 22:55:58.563389 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:58.562879    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" event={"ID":"e3de663c-a635-40f4-a536-ad32fcb3473a","Type":"ContainerDied","Data":"01f297cc82e513c1efd40d9b2c3bb9a307488245e908deed43e427626a8d093f"}
Oct 30 22:55:58.563389 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:58.562940    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx" event={"ID":"e3de663c-a635-40f4-a536-ad32fcb3473a","Type":"ContainerDied","Data":"aeda8d2df8939d45ae9893d385cda85663944882d508ba8737cadc3a071c50cd"}
Oct 30 22:55:58.563389 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:58.562967    4947 scope.go:122] "RemoveContainer" containerID="44f94044a1f2c322eeafea5ee1c148e7d7b365cf3ceb0a564a1ad7c4c9818667"
Oct 30 22:55:58.571372 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:58.571347    4947 scope.go:122] "RemoveContainer" containerID="01f297cc82e513c1efd40d9b2c3bb9a307488245e908deed43e427626a8d093f"
Oct 30 22:55:58.579112 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:58.579093    4947 scope.go:122] "RemoveContainer" containerID="beaf536ad1e3f9b58f8e233fbbce8bbf91983841dcc8f86c0847da419114632e"
Oct 30 22:55:58.586861 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:58.586839    4947 scope.go:122] "RemoveContainer" containerID="44f94044a1f2c322eeafea5ee1c148e7d7b365cf3ceb0a564a1ad7c4c9818667"
Oct 30 22:55:58.587137 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:55:58.587118    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f94044a1f2c322eeafea5ee1c148e7d7b365cf3ceb0a564a1ad7c4c9818667\": container with ID starting with 44f94044a1f2c322eeafea5ee1c148e7d7b365cf3ceb0a564a1ad7c4c9818667 not found: ID does not exist" containerID="44f94044a1f2c322eeafea5ee1c148e7d7b365cf3ceb0a564a1ad7c4c9818667"
Oct 30 22:55:58.587182 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:58.587148    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f94044a1f2c322eeafea5ee1c148e7d7b365cf3ceb0a564a1ad7c4c9818667"} err="failed to get container status \"44f94044a1f2c322eeafea5ee1c148e7d7b365cf3ceb0a564a1ad7c4c9818667\": rpc error: code = NotFound desc = could not find container \"44f94044a1f2c322eeafea5ee1c148e7d7b365cf3ceb0a564a1ad7c4c9818667\": container with ID starting with 44f94044a1f2c322eeafea5ee1c148e7d7b365cf3ceb0a564a1ad7c4c9818667 not found: ID does not exist"
Oct 30 22:55:58.587182 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:58.587169    4947 scope.go:122] "RemoveContainer" containerID="01f297cc82e513c1efd40d9b2c3bb9a307488245e908deed43e427626a8d093f"
Oct 30 22:55:58.587449 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:55:58.587430    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f297cc82e513c1efd40d9b2c3bb9a307488245e908deed43e427626a8d093f\": container with ID starting with 01f297cc82e513c1efd40d9b2c3bb9a307488245e908deed43e427626a8d093f not found: ID does not exist" containerID="01f297cc82e513c1efd40d9b2c3bb9a307488245e908deed43e427626a8d093f"
Oct 30 22:55:58.587494 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:58.587455    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f297cc82e513c1efd40d9b2c3bb9a307488245e908deed43e427626a8d093f"} err="failed to get container status \"01f297cc82e513c1efd40d9b2c3bb9a307488245e908deed43e427626a8d093f\": rpc error: code = NotFound desc = could not find container \"01f297cc82e513c1efd40d9b2c3bb9a307488245e908deed43e427626a8d093f\": container with ID starting with 01f297cc82e513c1efd40d9b2c3bb9a307488245e908deed43e427626a8d093f not found: ID does not exist"
Oct 30 22:55:58.587494 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:58.587471    4947 scope.go:122] "RemoveContainer" containerID="beaf536ad1e3f9b58f8e233fbbce8bbf91983841dcc8f86c0847da419114632e"
Oct 30 22:55:58.587692 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:55:58.587674    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beaf536ad1e3f9b58f8e233fbbce8bbf91983841dcc8f86c0847da419114632e\": container with ID starting with beaf536ad1e3f9b58f8e233fbbce8bbf91983841dcc8f86c0847da419114632e not found: ID does not exist" containerID="beaf536ad1e3f9b58f8e233fbbce8bbf91983841dcc8f86c0847da419114632e"
Oct 30 22:55:58.587730 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:58.587697    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beaf536ad1e3f9b58f8e233fbbce8bbf91983841dcc8f86c0847da419114632e"} err="failed to get container status \"beaf536ad1e3f9b58f8e233fbbce8bbf91983841dcc8f86c0847da419114632e\": rpc error: code = NotFound desc = could not find container \"beaf536ad1e3f9b58f8e233fbbce8bbf91983841dcc8f86c0847da419114632e\": container with ID starting with beaf536ad1e3f9b58f8e233fbbce8bbf91983841dcc8f86c0847da419114632e not found: ID does not exist"
Oct 30 22:55:58.780597 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:58.780567    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"]
Oct 30 22:55:58.830135 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:55:58.830059    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-xgboost-v2-predictor-7cfc9c5487-l69jx"]
Oct 30 22:55:59.048816 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:55:59.048785    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:56:00.157718 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:00.157683    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" path="/var/lib/kubelet/pods/e3de663c-a635-40f4-a536-ad32fcb3473a/volumes"
Oct 30 22:56:05.315842 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:05.315803    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 22:56:07.324090 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:07.324045    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 22:56:15.316205 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:15.316157    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 30 22:56:17.324940 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:17.324910    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:56:25.317277 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:25.317243    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:56:29.048527 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:56:29.048489    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:56:29.127299 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:29.127267    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"]
Oct 30 22:56:29.127571 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:29.127545    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kserve-container" containerID="cri-o://49cd188a37a556c254ae43011f97765d926eb611aa8e75cd5ad4070e94a61285" gracePeriod=30
Oct 30 22:56:29.127650 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:29.127607    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kube-rbac-proxy" containerID="cri-o://17800f07df0df030dbfcec665ec69e31a5fe6285f28ab42b3db061fc3a2634bd" gracePeriod=30
Oct 30 22:56:29.698118 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:29.698078    4947 generic.go:356] "Generic (PLEG): container finished" podID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerID="17800f07df0df030dbfcec665ec69e31a5fe6285f28ab42b3db061fc3a2634bd" exitCode=0
Oct 30 22:56:29.698398 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:29.698147    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" event={"ID":"93a9abf2-7c97-4999-960e-2005cdf94f34","Type":"ContainerDied","Data":"17800f07df0df030dbfcec665ec69e31a5fe6285f28ab42b3db061fc3a2634bd"}
Oct 30 22:56:30.311047 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:30.311003    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.66:8643/healthz\": dial tcp 10.132.0.66:8643: connect: connection refused"
Oct 30 22:56:33.877620 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:33.877596    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:56:34.050182 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.050150    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a9abf2-7c97-4999-960e-2005cdf94f34-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a9abf2-7c97-4999-960e-2005cdf94f34-cabundle-cert\") pod \"93a9abf2-7c97-4999-960e-2005cdf94f34\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") "
Oct 30 22:56:34.050395 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.050201    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a9abf2-7c97-4999-960e-2005cdf94f34-kube-api-access-85gtc\" (UniqueName: \"kubernetes.io/projected/93a9abf2-7c97-4999-960e-2005cdf94f34-kube-api-access-85gtc\") pod \"93a9abf2-7c97-4999-960e-2005cdf94f34\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") "
Oct 30 22:56:34.050395 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.050237    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a9abf2-7c97-4999-960e-2005cdf94f34-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a9abf2-7c97-4999-960e-2005cdf94f34-kube-rbac-proxy-sar-config\") pod \"93a9abf2-7c97-4999-960e-2005cdf94f34\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") "
Oct 30 22:56:34.050530 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.050426    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93a9abf2-7c97-4999-960e-2005cdf94f34-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a9abf2-7c97-4999-960e-2005cdf94f34-kserve-provision-location\") pod \"93a9abf2-7c97-4999-960e-2005cdf94f34\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") "
Oct 30 22:56:34.050588 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.050540    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93a9abf2-7c97-4999-960e-2005cdf94f34-proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a9abf2-7c97-4999-960e-2005cdf94f34-proxy-tls\") pod \"93a9abf2-7c97-4999-960e-2005cdf94f34\" (UID: \"93a9abf2-7c97-4999-960e-2005cdf94f34\") "
Oct 30 22:56:34.050647 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.050629    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a9abf2-7c97-4999-960e-2005cdf94f34-kube-rbac-proxy-sar-config" pod "93a9abf2-7c97-4999-960e-2005cdf94f34" (UID: "93a9abf2-7c97-4999-960e-2005cdf94f34"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:34.050727 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.050703    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a9abf2-7c97-4999-960e-2005cdf94f34-cabundle-cert" pod "93a9abf2-7c97-4999-960e-2005cdf94f34" (UID: "93a9abf2-7c97-4999-960e-2005cdf94f34"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:34.050768 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.050724    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a9abf2-7c97-4999-960e-2005cdf94f34-kserve-provision-location" pod "93a9abf2-7c97-4999-960e-2005cdf94f34" (UID: "93a9abf2-7c97-4999-960e-2005cdf94f34"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:34.050838 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.050819    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a9abf2-7c97-4999-960e-2005cdf94f34-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:34.050954 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.050841    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a9abf2-7c97-4999-960e-2005cdf94f34-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:34.050954 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.050855    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a9abf2-7c97-4999-960e-2005cdf94f34-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:34.052942 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.052917    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a9abf2-7c97-4999-960e-2005cdf94f34-kube-api-access-85gtc" pod "93a9abf2-7c97-4999-960e-2005cdf94f34" (UID: "93a9abf2-7c97-4999-960e-2005cdf94f34"). InnerVolumeSpecName "kube-api-access-85gtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:34.053027 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.052958    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a9abf2-7c97-4999-960e-2005cdf94f34-proxy-tls" pod "93a9abf2-7c97-4999-960e-2005cdf94f34" (UID: "93a9abf2-7c97-4999-960e-2005cdf94f34"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:34.151962 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.151929    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a9abf2-7c97-4999-960e-2005cdf94f34-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:34.151962 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.151956    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85gtc\" (UniqueName: \"kubernetes.io/projected/93a9abf2-7c97-4999-960e-2005cdf94f34-kube-api-access-85gtc\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:34.721581 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.721550    4947 generic.go:356] "Generic (PLEG): container finished" podID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerID="49cd188a37a556c254ae43011f97765d926eb611aa8e75cd5ad4070e94a61285" exitCode=0
Oct 30 22:56:34.721750 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.721606    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"
Oct 30 22:56:34.721750 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.721615    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" event={"ID":"93a9abf2-7c97-4999-960e-2005cdf94f34","Type":"ContainerDied","Data":"49cd188a37a556c254ae43011f97765d926eb611aa8e75cd5ad4070e94a61285"}
Oct 30 22:56:34.721750 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.721677    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5" event={"ID":"93a9abf2-7c97-4999-960e-2005cdf94f34","Type":"ContainerDied","Data":"525f3f3075819e932180773230f9f123599b3c6208f652c13c9c9cd03fe2b7b5"}
Oct 30 22:56:34.721750 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.721703    4947 scope.go:122] "RemoveContainer" containerID="17800f07df0df030dbfcec665ec69e31a5fe6285f28ab42b3db061fc3a2634bd"
Oct 30 22:56:34.730589 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.730573    4947 scope.go:122] "RemoveContainer" containerID="49cd188a37a556c254ae43011f97765d926eb611aa8e75cd5ad4070e94a61285"
Oct 30 22:56:34.738291 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.738273    4947 scope.go:122] "RemoveContainer" containerID="95cd5691924eb8cd1a734358290dadb32c2c0d4e981acb7d276ee3eb35d013eb"
Oct 30 22:56:34.747933 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.747724    4947 scope.go:122] "RemoveContainer" containerID="17800f07df0df030dbfcec665ec69e31a5fe6285f28ab42b3db061fc3a2634bd"
Oct 30 22:56:34.748134 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:56:34.748114    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17800f07df0df030dbfcec665ec69e31a5fe6285f28ab42b3db061fc3a2634bd\": container with ID starting with 17800f07df0df030dbfcec665ec69e31a5fe6285f28ab42b3db061fc3a2634bd not found: ID does not exist" containerID="17800f07df0df030dbfcec665ec69e31a5fe6285f28ab42b3db061fc3a2634bd"
Oct 30 22:56:34.748190 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.748153    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17800f07df0df030dbfcec665ec69e31a5fe6285f28ab42b3db061fc3a2634bd"} err="failed to get container status \"17800f07df0df030dbfcec665ec69e31a5fe6285f28ab42b3db061fc3a2634bd\": rpc error: code = NotFound desc = could not find container \"17800f07df0df030dbfcec665ec69e31a5fe6285f28ab42b3db061fc3a2634bd\": container with ID starting with 17800f07df0df030dbfcec665ec69e31a5fe6285f28ab42b3db061fc3a2634bd not found: ID does not exist"
Oct 30 22:56:34.748190 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.748171    4947 scope.go:122] "RemoveContainer" containerID="49cd188a37a556c254ae43011f97765d926eb611aa8e75cd5ad4070e94a61285"
Oct 30 22:56:34.748545 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:56:34.748495    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49cd188a37a556c254ae43011f97765d926eb611aa8e75cd5ad4070e94a61285\": container with ID starting with 49cd188a37a556c254ae43011f97765d926eb611aa8e75cd5ad4070e94a61285 not found: ID does not exist" containerID="49cd188a37a556c254ae43011f97765d926eb611aa8e75cd5ad4070e94a61285"
Oct 30 22:56:34.748641 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.748559    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49cd188a37a556c254ae43011f97765d926eb611aa8e75cd5ad4070e94a61285"} err="failed to get container status \"49cd188a37a556c254ae43011f97765d926eb611aa8e75cd5ad4070e94a61285\": rpc error: code = NotFound desc = could not find container \"49cd188a37a556c254ae43011f97765d926eb611aa8e75cd5ad4070e94a61285\": container with ID starting with 49cd188a37a556c254ae43011f97765d926eb611aa8e75cd5ad4070e94a61285 not found: ID does not exist"
Oct 30 22:56:34.748641 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.748581    4947 scope.go:122] "RemoveContainer" containerID="95cd5691924eb8cd1a734358290dadb32c2c0d4e981acb7d276ee3eb35d013eb"
Oct 30 22:56:34.748881 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:56:34.748863    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95cd5691924eb8cd1a734358290dadb32c2c0d4e981acb7d276ee3eb35d013eb\": container with ID starting with 95cd5691924eb8cd1a734358290dadb32c2c0d4e981acb7d276ee3eb35d013eb not found: ID does not exist" containerID="95cd5691924eb8cd1a734358290dadb32c2c0d4e981acb7d276ee3eb35d013eb"
Oct 30 22:56:34.748931 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.748891    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95cd5691924eb8cd1a734358290dadb32c2c0d4e981acb7d276ee3eb35d013eb"} err="failed to get container status \"95cd5691924eb8cd1a734358290dadb32c2c0d4e981acb7d276ee3eb35d013eb\": rpc error: code = NotFound desc = could not find container \"95cd5691924eb8cd1a734358290dadb32c2c0d4e981acb7d276ee3eb35d013eb\": container with ID starting with 95cd5691924eb8cd1a734358290dadb32c2c0d4e981acb7d276ee3eb35d013eb not found: ID does not exist"
Oct 30 22:56:34.947749 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.947713    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"]
Oct 30 22:56:34.997378 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:34.997275    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-rollback-v2-predictor-7db9b79f6c-5q7w5"]
Oct 30 22:56:36.157841 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:56:36.157804    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" path="/var/lib/kubelet/pods/93a9abf2-7c97-4999-960e-2005cdf94f34/volumes"
Oct 30 22:56:59.047452 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:56:59.047413    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:57:29.048676 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:57:29.048636    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:57:30.056429 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.056385    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"]
Oct 30 22:57:30.056851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.056709    4947 cpu_manager.go:414] "containerMap: removing container" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kserve-container"
Oct 30 22:57:30.056851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.056720    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kserve-container"
Oct 30 22:57:30.056851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.056739    4947 cpu_manager.go:414] "containerMap: removing container" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kube-rbac-proxy"
Oct 30 22:57:30.056851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.056744    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kube-rbac-proxy"
Oct 30 22:57:30.056851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.056751    4947 cpu_manager.go:414] "containerMap: removing container" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="storage-initializer"
Oct 30 22:57:30.056851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.056756    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="storage-initializer"
Oct 30 22:57:30.056851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.056768    4947 cpu_manager.go:414] "containerMap: removing container" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kube-rbac-proxy"
Oct 30 22:57:30.056851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.056772    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kube-rbac-proxy"
Oct 30 22:57:30.056851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.056779    4947 cpu_manager.go:414] "containerMap: removing container" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="storage-initializer"
Oct 30 22:57:30.056851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.056783    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="storage-initializer"
Oct 30 22:57:30.056851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.056797    4947 cpu_manager.go:414] "containerMap: removing container" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kserve-container"
Oct 30 22:57:30.056851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.056802    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kserve-container"
Oct 30 22:57:30.057254 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.056865    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kube-rbac-proxy"
Oct 30 22:57:30.057254 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.056874    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a9abf2-7c97-4999-960e-2005cdf94f34" containerName="kserve-container"
Oct 30 22:57:30.057254 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.056880    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kserve-container"
Oct 30 22:57:30.057254 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.056887    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3de663c-a635-40f4-a536-ad32fcb3473a" containerName="kube-rbac-proxy"
Oct 30 22:57:30.061147 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.061131    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:30.166486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.166412    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:57:30.166486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.166439    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:57:30.166486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.166453    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:57:30.166486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.166439    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"storage-config\""
Oct 30 22:57:30.166486 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.166476    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Oct 30 22:57:30.166786 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.166531    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Oct 30 22:57:30.166786 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.166592    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:57:30.168617 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.168594    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"]
Oct 30 22:57:30.187806 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.187779    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9kwm\" (UniqueName: \"kubernetes.io/projected/e374022d-8548-43bf-8891-e9de1d1f2fb8-kube-api-access-p9kwm\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:30.187920 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.187824    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e374022d-8548-43bf-8891-e9de1d1f2fb8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:30.187920 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.187845    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e374022d-8548-43bf-8891-e9de1d1f2fb8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:30.187995 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.187942    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e374022d-8548-43bf-8891-e9de1d1f2fb8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:30.188032 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.187995    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e374022d-8548-43bf-8891-e9de1d1f2fb8-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:30.288600 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.288568    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9kwm\" (UniqueName: \"kubernetes.io/projected/e374022d-8548-43bf-8891-e9de1d1f2fb8-kube-api-access-p9kwm\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:30.288787 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.288616    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e374022d-8548-43bf-8891-e9de1d1f2fb8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:30.288787 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.288634    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e374022d-8548-43bf-8891-e9de1d1f2fb8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:30.288787 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.288663    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e374022d-8548-43bf-8891-e9de1d1f2fb8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:30.288787 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.288699    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e374022d-8548-43bf-8891-e9de1d1f2fb8-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:30.289127 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.289104    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e374022d-8548-43bf-8891-e9de1d1f2fb8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:30.289314 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.289294    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e374022d-8548-43bf-8891-e9de1d1f2fb8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:30.289499 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.289478    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e374022d-8548-43bf-8891-e9de1d1f2fb8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:30.291364 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.291347    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e374022d-8548-43bf-8891-e9de1d1f2fb8-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:30.351545 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.351483    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9kwm\" (UniqueName: \"kubernetes.io/projected/e374022d-8548-43bf-8891-e9de1d1f2fb8-kube-api-access-p9kwm\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:30.378019 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.377990    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:30.803191 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.803148    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"]
Oct 30 22:57:30.807861 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:57:30.807835    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode374022d_8548_43bf_8891_e9de1d1f2fb8.slice/crio-12eadef9f6b3a0537738cbd6be9ebd0edc708a15029483fb10143eea69247fb8 WatchSource:0}: Error finding container 12eadef9f6b3a0537738cbd6be9ebd0edc708a15029483fb10143eea69247fb8: Status 404 returned error can't find the container with id 12eadef9f6b3a0537738cbd6be9ebd0edc708a15029483fb10143eea69247fb8
Oct 30 22:57:30.954771 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.954728    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" event={"ID":"e374022d-8548-43bf-8891-e9de1d1f2fb8","Type":"ContainerStarted","Data":"07d32bac3e8fb6ff9f13a1222e204bab7bc5e80e65f0792247bfb4db8816eac6"}
Oct 30 22:57:30.954943 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:30.954798    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" event={"ID":"e374022d-8548-43bf-8891-e9de1d1f2fb8","Type":"ContainerStarted","Data":"12eadef9f6b3a0537738cbd6be9ebd0edc708a15029483fb10143eea69247fb8"}
Oct 30 22:57:32.965293 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:32.965245    4947 generic.go:356] "Generic (PLEG): container finished" podID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerID="07d32bac3e8fb6ff9f13a1222e204bab7bc5e80e65f0792247bfb4db8816eac6" exitCode=0
Oct 30 22:57:32.965702 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:32.965316    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" event={"ID":"e374022d-8548-43bf-8891-e9de1d1f2fb8","Type":"ContainerDied","Data":"07d32bac3e8fb6ff9f13a1222e204bab7bc5e80e65f0792247bfb4db8816eac6"}
Oct 30 22:57:33.972150 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:33.972107    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" event={"ID":"e374022d-8548-43bf-8891-e9de1d1f2fb8","Type":"ContainerStarted","Data":"6251a355226f112fb1f3c4047e9289651f3073607cd01e669875b62ab27cc65f"}
Oct 30 22:57:33.972667 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:33.972187    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" event={"ID":"e374022d-8548-43bf-8891-e9de1d1f2fb8","Type":"ContainerStarted","Data":"5a6b58cc1a4931890d242cb073873895a3463eba48372119042da5470ebd4798"}
Oct 30 22:57:33.972667 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:33.972526    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:34.100113 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:34.100048    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" podStartSLOduration=2.936198095 podStartE2EDuration="4.100034121s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.163836026s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:34.098661721 +0000 UTC m=+2952.098009284" watchObservedRunningTime="2026-10-30 22:57:34.100034121 +0000 UTC m=+2952.099381681"
Oct 30 22:57:34.977679 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:34.977647    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:34.978811 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:34.978783    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 30 22:57:35.982684 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:35.982642    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 30 22:57:40.986796 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:40.986768    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:57:40.987415 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:40.987387    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 30 22:57:50.987424 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:50.987383    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 30 22:57:59.012417 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:59.012376    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"]
Oct 30 22:57:59.012816 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:59.012637    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kserve-container" containerID="cri-o://a8a3f7587bbd8b86fbae8744a3792a1ce550deb78828b81ceb219aa6c2ad59f3" gracePeriod=30
Oct 30 22:57:59.012816 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:57:59.012668    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kube-rbac-proxy" containerID="cri-o://f0dd2e1711016def728b684172999320125fd7b8bb3f29bce0d8d1445c064493" gracePeriod=30
Oct 30 22:57:59.047915 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:57:59.047883    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:58:00.084520 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:00.084485    4947 generic.go:356] "Generic (PLEG): container finished" podID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerID="f0dd2e1711016def728b684172999320125fd7b8bb3f29bce0d8d1445c064493" exitCode=0
Oct 30 22:58:00.084915 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:00.084557    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" event={"ID":"d5ddbfd1-213f-44ee-9b05-ec6477bac135","Type":"ContainerDied","Data":"f0dd2e1711016def728b684172999320125fd7b8bb3f29bce0d8d1445c064493"}
Oct 30 22:58:00.987722 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:00.987686    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 30 22:58:02.319064 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:02.319025    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.67:8643/healthz\": dial tcp 10.132.0.67:8643: connect: connection refused"
Oct 30 22:58:03.755715 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:03.755691    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:58:03.763623 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:03.763603    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5ddbfd1-213f-44ee-9b05-ec6477bac135-proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5ddbfd1-213f-44ee-9b05-ec6477bac135-proxy-tls\") pod \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") "
Oct 30 22:58:03.763703 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:03.763652    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5ddbfd1-213f-44ee-9b05-ec6477bac135-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5ddbfd1-213f-44ee-9b05-ec6477bac135-cabundle-cert\") pod \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") "
Oct 30 22:58:03.763703 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:03.763694    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kserve-provision-location\") pod \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") "
Oct 30 22:58:03.763821 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:03.763714    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kube-rbac-proxy-sar-config\") pod \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") "
Oct 30 22:58:03.763821 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:03.763741    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kube-api-access-5p7jz\" (UniqueName: \"kubernetes.io/projected/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kube-api-access-5p7jz\") pod \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\" (UID: \"d5ddbfd1-213f-44ee-9b05-ec6477bac135\") "
Oct 30 22:58:03.764086 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:03.764059    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kserve-provision-location" pod "d5ddbfd1-213f-44ee-9b05-ec6477bac135" (UID: "d5ddbfd1-213f-44ee-9b05-ec6477bac135"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:03.764148 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:03.764101    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kube-rbac-proxy-sar-config" pod "d5ddbfd1-213f-44ee-9b05-ec6477bac135" (UID: "d5ddbfd1-213f-44ee-9b05-ec6477bac135"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:58:03.764148 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:03.764106    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ddbfd1-213f-44ee-9b05-ec6477bac135-cabundle-cert" pod "d5ddbfd1-213f-44ee-9b05-ec6477bac135" (UID: "d5ddbfd1-213f-44ee-9b05-ec6477bac135"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:58:03.765729 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:03.765707    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5ddbfd1-213f-44ee-9b05-ec6477bac135-proxy-tls" pod "d5ddbfd1-213f-44ee-9b05-ec6477bac135" (UID: "d5ddbfd1-213f-44ee-9b05-ec6477bac135"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:03.765944 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:03.765925    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kube-api-access-5p7jz" pod "d5ddbfd1-213f-44ee-9b05-ec6477bac135" (UID: "d5ddbfd1-213f-44ee-9b05-ec6477bac135"). InnerVolumeSpecName "kube-api-access-5p7jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:03.864832 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:03.864759    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:58:03.864832 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:03.864782    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:58:03.864832 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:03.864792    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p7jz\" (UniqueName: \"kubernetes.io/projected/d5ddbfd1-213f-44ee-9b05-ec6477bac135-kube-api-access-5p7jz\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:58:03.864832 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:03.864801    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5ddbfd1-213f-44ee-9b05-ec6477bac135-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:58:03.864832 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:03.864810    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5ddbfd1-213f-44ee-9b05-ec6477bac135-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:58:04.103126 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:04.103088    4947 generic.go:356] "Generic (PLEG): container finished" podID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerID="a8a3f7587bbd8b86fbae8744a3792a1ce550deb78828b81ceb219aa6c2ad59f3" exitCode=0
Oct 30 22:58:04.103294 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:04.103158    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"
Oct 30 22:58:04.103294 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:04.103161    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" event={"ID":"d5ddbfd1-213f-44ee-9b05-ec6477bac135","Type":"ContainerDied","Data":"a8a3f7587bbd8b86fbae8744a3792a1ce550deb78828b81ceb219aa6c2ad59f3"}
Oct 30 22:58:04.103294 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:04.103259    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx" event={"ID":"d5ddbfd1-213f-44ee-9b05-ec6477bac135","Type":"ContainerDied","Data":"6c8c094f25a7833af0525a2ce536a15518b3f5ff46fb32017bea4615b17d7244"}
Oct 30 22:58:04.103460 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:04.103299    4947 scope.go:122] "RemoveContainer" containerID="f0dd2e1711016def728b684172999320125fd7b8bb3f29bce0d8d1445c064493"
Oct 30 22:58:04.112471 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:04.112448    4947 scope.go:122] "RemoveContainer" containerID="a8a3f7587bbd8b86fbae8744a3792a1ce550deb78828b81ceb219aa6c2ad59f3"
Oct 30 22:58:04.119577 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:04.119561    4947 scope.go:122] "RemoveContainer" containerID="234b818c2353f00dc29761676120efe8a13c4e79a67333c8c2277caa006a3570"
Oct 30 22:58:04.126817 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:04.126797    4947 scope.go:122] "RemoveContainer" containerID="f0dd2e1711016def728b684172999320125fd7b8bb3f29bce0d8d1445c064493"
Oct 30 22:58:04.127077 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:58:04.127060    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0dd2e1711016def728b684172999320125fd7b8bb3f29bce0d8d1445c064493\": container with ID starting with f0dd2e1711016def728b684172999320125fd7b8bb3f29bce0d8d1445c064493 not found: ID does not exist" containerID="f0dd2e1711016def728b684172999320125fd7b8bb3f29bce0d8d1445c064493"
Oct 30 22:58:04.127136 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:04.127085    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0dd2e1711016def728b684172999320125fd7b8bb3f29bce0d8d1445c064493"} err="failed to get container status \"f0dd2e1711016def728b684172999320125fd7b8bb3f29bce0d8d1445c064493\": rpc error: code = NotFound desc = could not find container \"f0dd2e1711016def728b684172999320125fd7b8bb3f29bce0d8d1445c064493\": container with ID starting with f0dd2e1711016def728b684172999320125fd7b8bb3f29bce0d8d1445c064493 not found: ID does not exist"
Oct 30 22:58:04.127136 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:04.127103    4947 scope.go:122] "RemoveContainer" containerID="a8a3f7587bbd8b86fbae8744a3792a1ce550deb78828b81ceb219aa6c2ad59f3"
Oct 30 22:58:04.127323 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:58:04.127304    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a3f7587bbd8b86fbae8744a3792a1ce550deb78828b81ceb219aa6c2ad59f3\": container with ID starting with a8a3f7587bbd8b86fbae8744a3792a1ce550deb78828b81ceb219aa6c2ad59f3 not found: ID does not exist" containerID="a8a3f7587bbd8b86fbae8744a3792a1ce550deb78828b81ceb219aa6c2ad59f3"
Oct 30 22:58:04.127422 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:04.127342    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a3f7587bbd8b86fbae8744a3792a1ce550deb78828b81ceb219aa6c2ad59f3"} err="failed to get container status \"a8a3f7587bbd8b86fbae8744a3792a1ce550deb78828b81ceb219aa6c2ad59f3\": rpc error: code = NotFound desc = could not find container \"a8a3f7587bbd8b86fbae8744a3792a1ce550deb78828b81ceb219aa6c2ad59f3\": container with ID starting with a8a3f7587bbd8b86fbae8744a3792a1ce550deb78828b81ceb219aa6c2ad59f3 not found: ID does not exist"
Oct 30 22:58:04.127422 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:04.127358    4947 scope.go:122] "RemoveContainer" containerID="234b818c2353f00dc29761676120efe8a13c4e79a67333c8c2277caa006a3570"
Oct 30 22:58:04.127570 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:58:04.127549    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"234b818c2353f00dc29761676120efe8a13c4e79a67333c8c2277caa006a3570\": container with ID starting with 234b818c2353f00dc29761676120efe8a13c4e79a67333c8c2277caa006a3570 not found: ID does not exist" containerID="234b818c2353f00dc29761676120efe8a13c4e79a67333c8c2277caa006a3570"
Oct 30 22:58:04.127604 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:04.127577    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"234b818c2353f00dc29761676120efe8a13c4e79a67333c8c2277caa006a3570"} err="failed to get container status \"234b818c2353f00dc29761676120efe8a13c4e79a67333c8c2277caa006a3570\": rpc error: code = NotFound desc = could not find container \"234b818c2353f00dc29761676120efe8a13c4e79a67333c8c2277caa006a3570\": container with ID starting with 234b818c2353f00dc29761676120efe8a13c4e79a67333c8c2277caa006a3570 not found: ID does not exist"
Oct 30 22:58:04.270821 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:04.270790    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"]
Oct 30 22:58:04.276079 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:04.276052    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-rollback-predictor-5597787b4-4vghx"]
Oct 30 22:58:06.158269 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:06.158237    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" path="/var/lib/kubelet/pods/d5ddbfd1-213f-44ee-9b05-ec6477bac135/volumes"
Oct 30 22:58:08.169155 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.169126    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"]
Oct 30 22:58:08.169541 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.169483    4947 cpu_manager.go:414] "containerMap: removing container" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kube-rbac-proxy"
Oct 30 22:58:08.169541 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.169494    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kube-rbac-proxy"
Oct 30 22:58:08.169541 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.169519    4947 cpu_manager.go:414] "containerMap: removing container" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="storage-initializer"
Oct 30 22:58:08.169541 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.169524    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="storage-initializer"
Oct 30 22:58:08.169541 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.169531    4947 cpu_manager.go:414] "containerMap: removing container" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kserve-container"
Oct 30 22:58:08.169541 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.169536    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kserve-container"
Oct 30 22:58:08.169731 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.169594    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kube-rbac-proxy"
Oct 30 22:58:08.169731 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.169605    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ddbfd1-213f-44ee-9b05-ec6477bac135" containerName="kserve-container"
Oct 30 22:58:08.173925 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.173908    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:08.199870 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.199845    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f14a9890-8797-4f73-bd89-90c9235642a7-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-75759d648c-tdmkr\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:08.199870 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.199877    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24vq6\" (UniqueName: \"kubernetes.io/projected/f14a9890-8797-4f73-bd89-90c9235642a7-kube-api-access-24vq6\") pod \"isvc-canary-stop-v2-predictor-75759d648c-tdmkr\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:08.200090 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.199911    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f14a9890-8797-4f73-bd89-90c9235642a7-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-75759d648c-tdmkr\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:08.200090 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.200026    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f14a9890-8797-4f73-bd89-90c9235642a7-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-75759d648c-tdmkr\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:08.200090 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.200065    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f14a9890-8797-4f73-bd89-90c9235642a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-75759d648c-tdmkr\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:08.234227 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.234197    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Oct 30 22:58:08.283360 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.283288    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"seaweedfs-s3-creds\""
Oct 30 22:58:08.283360 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.283312    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"default-dockercfg-s9cqw\""
Oct 30 22:58:08.283360 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.283384    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"openshift-service-ca.crt\""
Oct 30 22:58:08.283360 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.283388    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"kube-root-ca.crt\""
Oct 30 22:58:08.283360 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.283400    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Oct 30 22:58:08.283360 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.283415    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:58:08.284960 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.284939    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"]
Oct 30 22:58:08.301010 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.300987    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f14a9890-8797-4f73-bd89-90c9235642a7-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-75759d648c-tdmkr\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:08.301088 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.301015    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f14a9890-8797-4f73-bd89-90c9235642a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-75759d648c-tdmkr\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:08.301088 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.301067    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f14a9890-8797-4f73-bd89-90c9235642a7-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-75759d648c-tdmkr\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:08.301179 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.301087    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24vq6\" (UniqueName: \"kubernetes.io/projected/f14a9890-8797-4f73-bd89-90c9235642a7-kube-api-access-24vq6\") pod \"isvc-canary-stop-v2-predictor-75759d648c-tdmkr\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:08.301179 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.301106    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f14a9890-8797-4f73-bd89-90c9235642a7-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-75759d648c-tdmkr\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:08.301606 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.301586    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f14a9890-8797-4f73-bd89-90c9235642a7-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-75759d648c-tdmkr\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:08.301855 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.301835    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f14a9890-8797-4f73-bd89-90c9235642a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-75759d648c-tdmkr\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:08.302088 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.301913    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f14a9890-8797-4f73-bd89-90c9235642a7-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-75759d648c-tdmkr\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:08.303879 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.303857    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f14a9890-8797-4f73-bd89-90c9235642a7-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-75759d648c-tdmkr\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:08.364449 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.364404    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24vq6\" (UniqueName: \"kubernetes.io/projected/f14a9890-8797-4f73-bd89-90c9235642a7-kube-api-access-24vq6\") pod \"isvc-canary-stop-v2-predictor-75759d648c-tdmkr\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:08.490542 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.490451    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:08.921660 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:08.921623    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"]
Oct 30 22:58:08.924669 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:58:08.924644    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14a9890_8797_4f73_bd89_90c9235642a7.slice/crio-beb59c80ca657b3293a034cb6d873fbf7236d46ed2ffeb1e5a53e22f46f2d63b WatchSource:0}: Error finding container beb59c80ca657b3293a034cb6d873fbf7236d46ed2ffeb1e5a53e22f46f2d63b: Status 404 returned error can't find the container with id beb59c80ca657b3293a034cb6d873fbf7236d46ed2ffeb1e5a53e22f46f2d63b
Oct 30 22:58:09.125976 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.125941    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" event={"ID":"f14a9890-8797-4f73-bd89-90c9235642a7","Type":"ContainerStarted","Data":"6e3b854a57a8e64a5cb40770b31bbd7aac480e3f01b0d2d9d633a467917c7a8d"}
Oct 30 22:58:09.126161 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.126025    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" event={"ID":"f14a9890-8797-4f73-bd89-90c9235642a7","Type":"ContainerStarted","Data":"beb59c80ca657b3293a034cb6d873fbf7236d46ed2ffeb1e5a53e22f46f2d63b"}
Oct 30 22:58:09.188685 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.188608    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"]
Oct 30 22:58:09.193079 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.193059    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:09.210055 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.210028    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20f94049-b2d3-4263-b790-f332bd7c1db6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-c7bb5549b-6v22x\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:09.210179 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.210061    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20f94049-b2d3-4263-b790-f332bd7c1db6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-c7bb5549b-6v22x\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:09.210261 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.210215    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20f94049-b2d3-4263-b790-f332bd7c1db6-cabundle-cert\") pod \"isvc-canary-stop-predictor-c7bb5549b-6v22x\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:09.210261 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.210251    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20f94049-b2d3-4263-b790-f332bd7c1db6-proxy-tls\") pod \"isvc-canary-stop-predictor-c7bb5549b-6v22x\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:09.210557 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.210527    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gstpv\" (UniqueName: \"kubernetes.io/projected/20f94049-b2d3-4263-b790-f332bd7c1db6-kube-api-access-gstpv\") pod \"isvc-canary-stop-predictor-c7bb5549b-6v22x\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:09.246839 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.246811    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-canary-stop-predictor-serving-cert\""
Oct 30 22:58:09.253669 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.253639    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"]
Oct 30 22:58:09.311444 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.311417    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20f94049-b2d3-4263-b790-f332bd7c1db6-cabundle-cert\") pod \"isvc-canary-stop-predictor-c7bb5549b-6v22x\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:09.311604 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.311448    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20f94049-b2d3-4263-b790-f332bd7c1db6-proxy-tls\") pod \"isvc-canary-stop-predictor-c7bb5549b-6v22x\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:09.311604 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.311488    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gstpv\" (UniqueName: \"kubernetes.io/projected/20f94049-b2d3-4263-b790-f332bd7c1db6-kube-api-access-gstpv\") pod \"isvc-canary-stop-predictor-c7bb5549b-6v22x\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:09.311604 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.311532    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20f94049-b2d3-4263-b790-f332bd7c1db6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-c7bb5549b-6v22x\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:09.311604 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.311562    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20f94049-b2d3-4263-b790-f332bd7c1db6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-c7bb5549b-6v22x\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:09.312028 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.312005    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20f94049-b2d3-4263-b790-f332bd7c1db6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-c7bb5549b-6v22x\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:09.312287 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.312224    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20f94049-b2d3-4263-b790-f332bd7c1db6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-c7bb5549b-6v22x\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:09.312287 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.312284    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20f94049-b2d3-4263-b790-f332bd7c1db6-cabundle-cert\") pod \"isvc-canary-stop-predictor-c7bb5549b-6v22x\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:09.314032 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.314012    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20f94049-b2d3-4263-b790-f332bd7c1db6-proxy-tls\") pod \"isvc-canary-stop-predictor-c7bb5549b-6v22x\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:09.374221 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.374191    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gstpv\" (UniqueName: \"kubernetes.io/projected/20f94049-b2d3-4263-b790-f332bd7c1db6-kube-api-access-gstpv\") pod \"isvc-canary-stop-predictor-c7bb5549b-6v22x\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") " pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:09.510898 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.510801    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:09.945388 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:09.945356    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"]
Oct 30 22:58:09.948216 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:58:09.948190    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f94049_b2d3_4263_b790_f332bd7c1db6.slice/crio-0bccfe90a5fefc586c70a8299e2c3b99523957d1c6b4949275798c2099838b3d WatchSource:0}: Error finding container 0bccfe90a5fefc586c70a8299e2c3b99523957d1c6b4949275798c2099838b3d: Status 404 returned error can't find the container with id 0bccfe90a5fefc586c70a8299e2c3b99523957d1c6b4949275798c2099838b3d
Oct 30 22:58:10.132812 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:10.132773    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" event={"ID":"20f94049-b2d3-4263-b790-f332bd7c1db6","Type":"ContainerStarted","Data":"d07d76c93b87ca33785f521773aae7ffd6c450e56088e8c6262be50e5f2f93ae"}
Oct 30 22:58:10.132987 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:10.132854    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" event={"ID":"20f94049-b2d3-4263-b790-f332bd7c1db6","Type":"ContainerStarted","Data":"0bccfe90a5fefc586c70a8299e2c3b99523957d1c6b4949275798c2099838b3d"}
Oct 30 22:58:10.987554 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:10.987519    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 30 22:58:13.146389 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:13.146354    4947 generic.go:356] "Generic (PLEG): container finished" podID="f14a9890-8797-4f73-bd89-90c9235642a7" containerID="6e3b854a57a8e64a5cb40770b31bbd7aac480e3f01b0d2d9d633a467917c7a8d" exitCode=0
Oct 30 22:58:13.146772 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:13.146430    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" event={"ID":"f14a9890-8797-4f73-bd89-90c9235642a7","Type":"ContainerDied","Data":"6e3b854a57a8e64a5cb40770b31bbd7aac480e3f01b0d2d9d633a467917c7a8d"}
Oct 30 22:58:14.152576 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:14.152546    4947 generic.go:356] "Generic (PLEG): container finished" podID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerID="d07d76c93b87ca33785f521773aae7ffd6c450e56088e8c6262be50e5f2f93ae" exitCode=0
Oct 30 22:58:14.163161 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:14.163132    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" event={"ID":"20f94049-b2d3-4263-b790-f332bd7c1db6","Type":"ContainerDied","Data":"d07d76c93b87ca33785f521773aae7ffd6c450e56088e8c6262be50e5f2f93ae"}
Oct 30 22:58:14.163273 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:14.163194    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" event={"ID":"f14a9890-8797-4f73-bd89-90c9235642a7","Type":"ContainerStarted","Data":"787eba837e8775675559b8fdf7e0edf3a9f89f857d0b6cf43837b22db2cab440"}
Oct 30 22:58:14.163273 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:14.163221    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" event={"ID":"f14a9890-8797-4f73-bd89-90c9235642a7","Type":"ContainerStarted","Data":"0eac5917fe06ba0eb858a93772c27ceaa557456440f8adbc5f56534bd88cd2da"}
Oct 30 22:58:14.163694 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:14.163669    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:14.163808 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:14.163721    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:14.164917 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:14.164877    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 22:58:14.303264 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:14.303206    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" podStartSLOduration=2.302897059 podStartE2EDuration="6.303188283s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.000291224s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:14.301372112 +0000 UTC m=+2992.300719676" watchObservedRunningTime="2026-10-30 22:58:14.303188283 +0000 UTC m=+2992.302535850"
Oct 30 22:58:15.161274 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:15.161235    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" event={"ID":"20f94049-b2d3-4263-b790-f332bd7c1db6","Type":"ContainerStarted","Data":"72bbd9272bfbddc7355d471072ca95dbc59ad531cfd8cdb9c53342b05850f932"}
Oct 30 22:58:15.161755 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:15.161302    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" event={"ID":"20f94049-b2d3-4263-b790-f332bd7c1db6","Type":"ContainerStarted","Data":"0286bb0c8f8d92553d21ee8c08149802dde924168ad4b0e642388e7191a6c65c"}
Oct 30 22:58:15.161755 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:15.161457    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 22:58:15.161755 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:15.161539    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:15.283963 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:15.283896    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" podStartSLOduration=2.17252188 podStartE2EDuration="6.283879027s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.111357147s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:15.282941592 +0000 UTC m=+2993.282289156" watchObservedRunningTime="2026-10-30 22:58:15.283879027 +0000 UTC m=+2993.283226589"
Oct 30 22:58:16.164995 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:16.164953    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 22:58:16.165494 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:16.165249    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:16.166248 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:16.166222    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 22:58:17.169415 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:17.169371    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 22:58:20.987645 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:20.987604    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 30 22:58:21.169856 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:21.169829    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:58:21.170311 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:21.170288    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 22:58:22.174134 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:22.174102    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:58:22.174529 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:22.174477    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 22:58:22.420668 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:22.420641    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:58:22.426053 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:22.425996    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 22:58:29.050556 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:58:29.050522    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:58:30.988384 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:30.988325    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 30 22:58:31.170583 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:31.170544    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 22:58:32.174794 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:32.174701    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 22:58:40.987508 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:40.987467    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 30 22:58:41.171301 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:41.171258    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 22:58:42.174568 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:42.174532    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 22:58:50.987510 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:50.987479    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:58:51.170518 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:51.170481    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 22:58:52.174574 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:52.174530    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 22:58:59.048686 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:58:59.048648    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:58:59.990247 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:59.990209    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"]
Oct 30 22:58:59.990510 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:59.990469    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kserve-container" containerID="cri-o://5a6b58cc1a4931890d242cb073873895a3463eba48372119042da5470ebd4798" gracePeriod=30
Oct 30 22:58:59.990586 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:58:59.990524    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kube-rbac-proxy" containerID="cri-o://6251a355226f112fb1f3c4047e9289651f3073607cd01e669875b62ab27cc65f" gracePeriod=30
Oct 30 22:59:00.357904 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:00.357871    4947 generic.go:356] "Generic (PLEG): container finished" podID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerID="6251a355226f112fb1f3c4047e9289651f3073607cd01e669875b62ab27cc65f" exitCode=0
Oct 30 22:59:00.358290 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:00.357940    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" event={"ID":"e374022d-8548-43bf-8891-e9de1d1f2fb8","Type":"ContainerDied","Data":"6251a355226f112fb1f3c4047e9289651f3073607cd01e669875b62ab27cc65f"}
Oct 30 22:59:00.983496 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:00.983458    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.68:8643/healthz\": dial tcp 10.132.0.68:8643: connect: connection refused"
Oct 30 22:59:00.987776 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:00.987754    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 30 22:59:01.170402 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:01.170367    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 22:59:02.175395 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:02.175363    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 22:59:05.236599 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.236573    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:59:05.291061 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.291035    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e374022d-8548-43bf-8891-e9de1d1f2fb8-kube-api-access-p9kwm\" (UniqueName: \"kubernetes.io/projected/e374022d-8548-43bf-8891-e9de1d1f2fb8-kube-api-access-p9kwm\") pod \"e374022d-8548-43bf-8891-e9de1d1f2fb8\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") "
Oct 30 22:59:05.291194 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.291077    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e374022d-8548-43bf-8891-e9de1d1f2fb8-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e374022d-8548-43bf-8891-e9de1d1f2fb8-cabundle-cert\") pod \"e374022d-8548-43bf-8891-e9de1d1f2fb8\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") "
Oct 30 22:59:05.291194 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.291135    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e374022d-8548-43bf-8891-e9de1d1f2fb8-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e374022d-8548-43bf-8891-e9de1d1f2fb8-kserve-provision-location\") pod \"e374022d-8548-43bf-8891-e9de1d1f2fb8\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") "
Oct 30 22:59:05.291194 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.291152    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e374022d-8548-43bf-8891-e9de1d1f2fb8-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e374022d-8548-43bf-8891-e9de1d1f2fb8-proxy-tls\") pod \"e374022d-8548-43bf-8891-e9de1d1f2fb8\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") "
Oct 30 22:59:05.291311 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.291225    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e374022d-8548-43bf-8891-e9de1d1f2fb8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e374022d-8548-43bf-8891-e9de1d1f2fb8-kube-rbac-proxy-sar-config\") pod \"e374022d-8548-43bf-8891-e9de1d1f2fb8\" (UID: \"e374022d-8548-43bf-8891-e9de1d1f2fb8\") "
Oct 30 22:59:05.291508 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.291486    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e374022d-8548-43bf-8891-e9de1d1f2fb8-kserve-provision-location" pod "e374022d-8548-43bf-8891-e9de1d1f2fb8" (UID: "e374022d-8548-43bf-8891-e9de1d1f2fb8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:59:05.291573 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.291517    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e374022d-8548-43bf-8891-e9de1d1f2fb8-cabundle-cert" pod "e374022d-8548-43bf-8891-e9de1d1f2fb8" (UID: "e374022d-8548-43bf-8891-e9de1d1f2fb8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:59:05.291709 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.291688    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e374022d-8548-43bf-8891-e9de1d1f2fb8-kube-rbac-proxy-sar-config" pod "e374022d-8548-43bf-8891-e9de1d1f2fb8" (UID: "e374022d-8548-43bf-8891-e9de1d1f2fb8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:59:05.293379 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.293358    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e374022d-8548-43bf-8891-e9de1d1f2fb8-kube-api-access-p9kwm" pod "e374022d-8548-43bf-8891-e9de1d1f2fb8" (UID: "e374022d-8548-43bf-8891-e9de1d1f2fb8"). InnerVolumeSpecName "kube-api-access-p9kwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:59:05.293472 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.293442    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e374022d-8548-43bf-8891-e9de1d1f2fb8-proxy-tls" pod "e374022d-8548-43bf-8891-e9de1d1f2fb8" (UID: "e374022d-8548-43bf-8891-e9de1d1f2fb8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:59:05.381634 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.381598    4947 generic.go:356] "Generic (PLEG): container finished" podID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerID="5a6b58cc1a4931890d242cb073873895a3463eba48372119042da5470ebd4798" exitCode=0
Oct 30 22:59:05.381814 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.381667    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"
Oct 30 22:59:05.381814 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.381687    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" event={"ID":"e374022d-8548-43bf-8891-e9de1d1f2fb8","Type":"ContainerDied","Data":"5a6b58cc1a4931890d242cb073873895a3463eba48372119042da5470ebd4798"}
Oct 30 22:59:05.381814 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.381755    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m" event={"ID":"e374022d-8548-43bf-8891-e9de1d1f2fb8","Type":"ContainerDied","Data":"12eadef9f6b3a0537738cbd6be9ebd0edc708a15029483fb10143eea69247fb8"}
Oct 30 22:59:05.381814 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.381794    4947 scope.go:122] "RemoveContainer" containerID="6251a355226f112fb1f3c4047e9289651f3073607cd01e669875b62ab27cc65f"
Oct 30 22:59:05.390718 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.390696    4947 scope.go:122] "RemoveContainer" containerID="5a6b58cc1a4931890d242cb073873895a3463eba48372119042da5470ebd4798"
Oct 30 22:59:05.391833 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.391814    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e374022d-8548-43bf-8891-e9de1d1f2fb8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:59:05.391915 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.391838    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9kwm\" (UniqueName: \"kubernetes.io/projected/e374022d-8548-43bf-8891-e9de1d1f2fb8-kube-api-access-p9kwm\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:59:05.391915 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.391855    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e374022d-8548-43bf-8891-e9de1d1f2fb8-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:59:05.391915 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.391869    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e374022d-8548-43bf-8891-e9de1d1f2fb8-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:59:05.391915 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.391882    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e374022d-8548-43bf-8891-e9de1d1f2fb8-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:59:05.398626 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.398596    4947 scope.go:122] "RemoveContainer" containerID="07d32bac3e8fb6ff9f13a1222e204bab7bc5e80e65f0792247bfb4db8816eac6"
Oct 30 22:59:05.407842 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.407821    4947 scope.go:122] "RemoveContainer" containerID="6251a355226f112fb1f3c4047e9289651f3073607cd01e669875b62ab27cc65f"
Oct 30 22:59:05.408134 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:59:05.408117    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6251a355226f112fb1f3c4047e9289651f3073607cd01e669875b62ab27cc65f\": container with ID starting with 6251a355226f112fb1f3c4047e9289651f3073607cd01e669875b62ab27cc65f not found: ID does not exist" containerID="6251a355226f112fb1f3c4047e9289651f3073607cd01e669875b62ab27cc65f"
Oct 30 22:59:05.408209 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.408145    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6251a355226f112fb1f3c4047e9289651f3073607cd01e669875b62ab27cc65f"} err="failed to get container status \"6251a355226f112fb1f3c4047e9289651f3073607cd01e669875b62ab27cc65f\": rpc error: code = NotFound desc = could not find container \"6251a355226f112fb1f3c4047e9289651f3073607cd01e669875b62ab27cc65f\": container with ID starting with 6251a355226f112fb1f3c4047e9289651f3073607cd01e669875b62ab27cc65f not found: ID does not exist"
Oct 30 22:59:05.408209 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.408163    4947 scope.go:122] "RemoveContainer" containerID="5a6b58cc1a4931890d242cb073873895a3463eba48372119042da5470ebd4798"
Oct 30 22:59:05.408454 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:59:05.408422    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a6b58cc1a4931890d242cb073873895a3463eba48372119042da5470ebd4798\": container with ID starting with 5a6b58cc1a4931890d242cb073873895a3463eba48372119042da5470ebd4798 not found: ID does not exist" containerID="5a6b58cc1a4931890d242cb073873895a3463eba48372119042da5470ebd4798"
Oct 30 22:59:05.408507 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.408460    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a6b58cc1a4931890d242cb073873895a3463eba48372119042da5470ebd4798"} err="failed to get container status \"5a6b58cc1a4931890d242cb073873895a3463eba48372119042da5470ebd4798\": rpc error: code = NotFound desc = could not find container \"5a6b58cc1a4931890d242cb073873895a3463eba48372119042da5470ebd4798\": container with ID starting with 5a6b58cc1a4931890d242cb073873895a3463eba48372119042da5470ebd4798 not found: ID does not exist"
Oct 30 22:59:05.408507 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.408476    4947 scope.go:122] "RemoveContainer" containerID="07d32bac3e8fb6ff9f13a1222e204bab7bc5e80e65f0792247bfb4db8816eac6"
Oct 30 22:59:05.408912 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:59:05.408754    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d32bac3e8fb6ff9f13a1222e204bab7bc5e80e65f0792247bfb4db8816eac6\": container with ID starting with 07d32bac3e8fb6ff9f13a1222e204bab7bc5e80e65f0792247bfb4db8816eac6 not found: ID does not exist" containerID="07d32bac3e8fb6ff9f13a1222e204bab7bc5e80e65f0792247bfb4db8816eac6"
Oct 30 22:59:05.408912 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.408788    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d32bac3e8fb6ff9f13a1222e204bab7bc5e80e65f0792247bfb4db8816eac6"} err="failed to get container status \"07d32bac3e8fb6ff9f13a1222e204bab7bc5e80e65f0792247bfb4db8816eac6\": rpc error: code = NotFound desc = could not find container \"07d32bac3e8fb6ff9f13a1222e204bab7bc5e80e65f0792247bfb4db8816eac6\": container with ID starting with 07d32bac3e8fb6ff9f13a1222e204bab7bc5e80e65f0792247bfb4db8816eac6 not found: ID does not exist"
Oct 30 22:59:05.523589 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.523549    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"]
Oct 30 22:59:05.529443 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:05.529417    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-global-pass-predictor-6c8557cc86-b7n6m"]
Oct 30 22:59:06.158887 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:06.158857    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" path="/var/lib/kubelet/pods/e374022d-8548-43bf-8891-e9de1d1f2fb8/volumes"
Oct 30 22:59:11.171143 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:11.171108    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 22:59:12.175196 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:12.175162    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 22:59:16.453277 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.453242    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"]
Oct 30 22:59:16.453746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.453589    4947 cpu_manager.go:414] "containerMap: removing container" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kserve-container"
Oct 30 22:59:16.453746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.453602    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kserve-container"
Oct 30 22:59:16.453746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.453625    4947 cpu_manager.go:414] "containerMap: removing container" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kube-rbac-proxy"
Oct 30 22:59:16.453746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.453630    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kube-rbac-proxy"
Oct 30 22:59:16.453746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.453642    4947 cpu_manager.go:414] "containerMap: removing container" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="storage-initializer"
Oct 30 22:59:16.453746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.453647    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="storage-initializer"
Oct 30 22:59:16.453746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.453697    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kserve-container"
Oct 30 22:59:16.453746 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.453706    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e374022d-8548-43bf-8891-e9de1d1f2fb8" containerName="kube-rbac-proxy"
Oct 30 22:59:16.458289 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.458270    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:16.512190 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.512163    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 22:59:16.561446 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.561423    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 22:59:16.561657 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.561638    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Oct 30 22:59:16.561742 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.561658    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:59:16.561888 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.561873    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Oct 30 22:59:16.561954 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.561903    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"storage-config\""
Oct 30 22:59:16.561954 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.561929    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 22:59:16.563392 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.563371    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"]
Oct 30 22:59:16.584258 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.584234    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8284975a-02bf-4d35-b6ea-28edaa42e622-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:16.584373 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.584266    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8284975a-02bf-4d35-b6ea-28edaa42e622-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:16.584373 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.584285    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6xx4\" (UniqueName: \"kubernetes.io/projected/8284975a-02bf-4d35-b6ea-28edaa42e622-kube-api-access-g6xx4\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:16.584456 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.584366    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8284975a-02bf-4d35-b6ea-28edaa42e622-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:16.584456 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.584425    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8284975a-02bf-4d35-b6ea-28edaa42e622-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:16.685824 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.685782    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8284975a-02bf-4d35-b6ea-28edaa42e622-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:16.686007 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.685843    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8284975a-02bf-4d35-b6ea-28edaa42e622-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:16.686007 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.685879    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6xx4\" (UniqueName: \"kubernetes.io/projected/8284975a-02bf-4d35-b6ea-28edaa42e622-kube-api-access-g6xx4\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:16.686007 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.685912    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8284975a-02bf-4d35-b6ea-28edaa42e622-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:16.686007 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.685953    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8284975a-02bf-4d35-b6ea-28edaa42e622-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:16.686383 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.686327    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8284975a-02bf-4d35-b6ea-28edaa42e622-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:16.686593 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.686571    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8284975a-02bf-4d35-b6ea-28edaa42e622-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:16.686652 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.686599    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8284975a-02bf-4d35-b6ea-28edaa42e622-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:16.688654 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.688633    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8284975a-02bf-4d35-b6ea-28edaa42e622-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:16.748775 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.748710    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6xx4\" (UniqueName: \"kubernetes.io/projected/8284975a-02bf-4d35-b6ea-28edaa42e622-kube-api-access-g6xx4\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:16.775537 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:16.775510    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:17.204842 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:17.204757    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"]
Oct 30 22:59:17.208877 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:59:17.208847    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8284975a_02bf_4d35_b6ea_28edaa42e622.slice/crio-dd7e5895e434740770df5ccd1b02c36df4cfe4133125ce91da47f715e3a72701 WatchSource:0}: Error finding container dd7e5895e434740770df5ccd1b02c36df4cfe4133125ce91da47f715e3a72701: Status 404 returned error can't find the container with id dd7e5895e434740770df5ccd1b02c36df4cfe4133125ce91da47f715e3a72701
Oct 30 22:59:17.434120 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:17.434075    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" event={"ID":"8284975a-02bf-4d35-b6ea-28edaa42e622","Type":"ContainerStarted","Data":"143329f5f41a5acfcfdd9eb9b1d5148588f6cc14d34f988741c97d0c9f650e1d"}
Oct 30 22:59:17.434298 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:17.434192    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" event={"ID":"8284975a-02bf-4d35-b6ea-28edaa42e622","Type":"ContainerStarted","Data":"dd7e5895e434740770df5ccd1b02c36df4cfe4133125ce91da47f715e3a72701"}
Oct 30 22:59:18.439212 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:18.439183    4947 generic.go:356] "Generic (PLEG): container finished" podID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerID="143329f5f41a5acfcfdd9eb9b1d5148588f6cc14d34f988741c97d0c9f650e1d" exitCode=0
Oct 30 22:59:18.439597 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:18.439219    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" event={"ID":"8284975a-02bf-4d35-b6ea-28edaa42e622","Type":"ContainerDied","Data":"143329f5f41a5acfcfdd9eb9b1d5148588f6cc14d34f988741c97d0c9f650e1d"}
Oct 30 22:59:19.446004 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:19.445971    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" event={"ID":"8284975a-02bf-4d35-b6ea-28edaa42e622","Type":"ContainerStarted","Data":"e4e46bbffaf6b611d112f56a81b23c72288d2b5c71190a62f0f2946bfbeecc4b"}
Oct 30 22:59:19.446484 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:19.446039    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" event={"ID":"8284975a-02bf-4d35-b6ea-28edaa42e622","Type":"ContainerStarted","Data":"a8358dcedd7ca0c88ec21d8c5dc0ee07be9824d88fd7aba0cfbb6f5557d755b4"}
Oct 30 22:59:19.446484 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:19.446149    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:19.446484 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:19.446293    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:19.447401 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:19.447375    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 22:59:19.571058 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:19.570999    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" podStartSLOduration=2.449341324 podStartE2EDuration="3.570984534s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.12164321s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:19.568967145 +0000 UTC m=+3057.568314707" watchObservedRunningTime="2026-10-30 22:59:19.570984534 +0000 UTC m=+3057.570332147"
Oct 30 22:59:20.449941 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:20.449906    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 22:59:21.170707 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:21.170668    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 22:59:22.174764 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:22.174723    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 22:59:22.605500 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:22.605466    4947 scope.go:122] "RemoveContainer" containerID="30ec07c9135266ee15c8c95efac97a28ddf5a70d376564b9706aa814ae59ab42"
Oct 30 22:59:25.454078 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:25.454048    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 22:59:25.454559 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:25.454534    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 22:59:27.153713 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:27.153659    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 22:59:29.048509 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:59:29.048469    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 22:59:31.171212 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:31.171175    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 30 22:59:33.154189 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:33.154156    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:59:34.180638 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:34.180601    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"]
Oct 30 22:59:34.181098 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:34.180827    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kserve-container" containerID="cri-o://0286bb0c8f8d92553d21ee8c08149802dde924168ad4b0e642388e7191a6c65c" gracePeriod=30
Oct 30 22:59:34.181098 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:34.180900    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kube-rbac-proxy" containerID="cri-o://72bbd9272bfbddc7355d471072ca95dbc59ad531cfd8cdb9c53342b05850f932" gracePeriod=30
Oct 30 22:59:34.181528 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:34.181507    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:59:34.511868 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:34.511789    4947 generic.go:356] "Generic (PLEG): container finished" podID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerID="72bbd9272bfbddc7355d471072ca95dbc59ad531cfd8cdb9c53342b05850f932" exitCode=0
Oct 30 22:59:34.512006 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:34.511861    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" event={"ID":"20f94049-b2d3-4263-b790-f332bd7c1db6","Type":"ContainerDied","Data":"72bbd9272bfbddc7355d471072ca95dbc59ad531cfd8cdb9c53342b05850f932"}
Oct 30 22:59:35.455068 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:35.455029    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 22:59:37.169851 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:37.169812    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.70:8643/healthz\": dial tcp 10.132.0.70:8643: connect: connection refused"
Oct 30 22:59:39.135185 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.135160    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:59:39.276724 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.276634    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f94049-b2d3-4263-b790-f332bd7c1db6-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20f94049-b2d3-4263-b790-f332bd7c1db6-kserve-provision-location\") pod \"20f94049-b2d3-4263-b790-f332bd7c1db6\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") "
Oct 30 22:59:39.276886 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.276739    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20f94049-b2d3-4263-b790-f332bd7c1db6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20f94049-b2d3-4263-b790-f332bd7c1db6-kube-rbac-proxy-sar-config\") pod \"20f94049-b2d3-4263-b790-f332bd7c1db6\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") "
Oct 30 22:59:39.276886 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.276811    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f94049-b2d3-4263-b790-f332bd7c1db6-kube-api-access-gstpv\" (UniqueName: \"kubernetes.io/projected/20f94049-b2d3-4263-b790-f332bd7c1db6-kube-api-access-gstpv\") pod \"20f94049-b2d3-4263-b790-f332bd7c1db6\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") "
Oct 30 22:59:39.276886 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.276860    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20f94049-b2d3-4263-b790-f332bd7c1db6-proxy-tls\" (UniqueName: \"kubernetes.io/secret/20f94049-b2d3-4263-b790-f332bd7c1db6-proxy-tls\") pod \"20f94049-b2d3-4263-b790-f332bd7c1db6\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") "
Oct 30 22:59:39.277053 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.276897    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20f94049-b2d3-4263-b790-f332bd7c1db6-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20f94049-b2d3-4263-b790-f332bd7c1db6-cabundle-cert\") pod \"20f94049-b2d3-4263-b790-f332bd7c1db6\" (UID: \"20f94049-b2d3-4263-b790-f332bd7c1db6\") "
Oct 30 22:59:39.277053 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.277004    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f94049-b2d3-4263-b790-f332bd7c1db6-kserve-provision-location" pod "20f94049-b2d3-4263-b790-f332bd7c1db6" (UID: "20f94049-b2d3-4263-b790-f332bd7c1db6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:59:39.277161 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.277098    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f94049-b2d3-4263-b790-f332bd7c1db6-kube-rbac-proxy-sar-config" pod "20f94049-b2d3-4263-b790-f332bd7c1db6" (UID: "20f94049-b2d3-4263-b790-f332bd7c1db6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:59:39.277269 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.277251    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20f94049-b2d3-4263-b790-f332bd7c1db6-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:59:39.277404 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.277383    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f94049-b2d3-4263-b790-f332bd7c1db6-cabundle-cert" pod "20f94049-b2d3-4263-b790-f332bd7c1db6" (UID: "20f94049-b2d3-4263-b790-f332bd7c1db6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:59:39.279053 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.279024    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f94049-b2d3-4263-b790-f332bd7c1db6-kube-api-access-gstpv" pod "20f94049-b2d3-4263-b790-f332bd7c1db6" (UID: "20f94049-b2d3-4263-b790-f332bd7c1db6"). InnerVolumeSpecName "kube-api-access-gstpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:59:39.279164 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.279069    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f94049-b2d3-4263-b790-f332bd7c1db6-proxy-tls" pod "20f94049-b2d3-4263-b790-f332bd7c1db6" (UID: "20f94049-b2d3-4263-b790-f332bd7c1db6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:59:39.378040 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.378006    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20f94049-b2d3-4263-b790-f332bd7c1db6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:59:39.378040 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.378037    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gstpv\" (UniqueName: \"kubernetes.io/projected/20f94049-b2d3-4263-b790-f332bd7c1db6-kube-api-access-gstpv\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:59:39.378040 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.378048    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20f94049-b2d3-4263-b790-f332bd7c1db6-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:59:39.378276 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.378057    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20f94049-b2d3-4263-b790-f332bd7c1db6-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:59:39.534458 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.534372    4947 generic.go:356] "Generic (PLEG): container finished" podID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerID="0286bb0c8f8d92553d21ee8c08149802dde924168ad4b0e642388e7191a6c65c" exitCode=0
Oct 30 22:59:39.534458 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.534430    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"
Oct 30 22:59:39.534649 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.534457    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" event={"ID":"20f94049-b2d3-4263-b790-f332bd7c1db6","Type":"ContainerDied","Data":"0286bb0c8f8d92553d21ee8c08149802dde924168ad4b0e642388e7191a6c65c"}
Oct 30 22:59:39.534649 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.534513    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x" event={"ID":"20f94049-b2d3-4263-b790-f332bd7c1db6","Type":"ContainerDied","Data":"0bccfe90a5fefc586c70a8299e2c3b99523957d1c6b4949275798c2099838b3d"}
Oct 30 22:59:39.534649 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.534540    4947 scope.go:122] "RemoveContainer" containerID="72bbd9272bfbddc7355d471072ca95dbc59ad531cfd8cdb9c53342b05850f932"
Oct 30 22:59:39.543682 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.543666    4947 scope.go:122] "RemoveContainer" containerID="0286bb0c8f8d92553d21ee8c08149802dde924168ad4b0e642388e7191a6c65c"
Oct 30 22:59:39.551325 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.551309    4947 scope.go:122] "RemoveContainer" containerID="d07d76c93b87ca33785f521773aae7ffd6c450e56088e8c6262be50e5f2f93ae"
Oct 30 22:59:39.558820 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.558798    4947 scope.go:122] "RemoveContainer" containerID="72bbd9272bfbddc7355d471072ca95dbc59ad531cfd8cdb9c53342b05850f932"
Oct 30 22:59:39.559105 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:59:39.559081    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72bbd9272bfbddc7355d471072ca95dbc59ad531cfd8cdb9c53342b05850f932\": container with ID starting with 72bbd9272bfbddc7355d471072ca95dbc59ad531cfd8cdb9c53342b05850f932 not found: ID does not exist" containerID="72bbd9272bfbddc7355d471072ca95dbc59ad531cfd8cdb9c53342b05850f932"
Oct 30 22:59:39.559234 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.559116    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72bbd9272bfbddc7355d471072ca95dbc59ad531cfd8cdb9c53342b05850f932"} err="failed to get container status \"72bbd9272bfbddc7355d471072ca95dbc59ad531cfd8cdb9c53342b05850f932\": rpc error: code = NotFound desc = could not find container \"72bbd9272bfbddc7355d471072ca95dbc59ad531cfd8cdb9c53342b05850f932\": container with ID starting with 72bbd9272bfbddc7355d471072ca95dbc59ad531cfd8cdb9c53342b05850f932 not found: ID does not exist"
Oct 30 22:59:39.559234 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.559142    4947 scope.go:122] "RemoveContainer" containerID="0286bb0c8f8d92553d21ee8c08149802dde924168ad4b0e642388e7191a6c65c"
Oct 30 22:59:39.559533 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:59:39.559510    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0286bb0c8f8d92553d21ee8c08149802dde924168ad4b0e642388e7191a6c65c\": container with ID starting with 0286bb0c8f8d92553d21ee8c08149802dde924168ad4b0e642388e7191a6c65c not found: ID does not exist" containerID="0286bb0c8f8d92553d21ee8c08149802dde924168ad4b0e642388e7191a6c65c"
Oct 30 22:59:39.559609 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.559541    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0286bb0c8f8d92553d21ee8c08149802dde924168ad4b0e642388e7191a6c65c"} err="failed to get container status \"0286bb0c8f8d92553d21ee8c08149802dde924168ad4b0e642388e7191a6c65c\": rpc error: code = NotFound desc = could not find container \"0286bb0c8f8d92553d21ee8c08149802dde924168ad4b0e642388e7191a6c65c\": container with ID starting with 0286bb0c8f8d92553d21ee8c08149802dde924168ad4b0e642388e7191a6c65c not found: ID does not exist"
Oct 30 22:59:39.559609 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.559559    4947 scope.go:122] "RemoveContainer" containerID="d07d76c93b87ca33785f521773aae7ffd6c450e56088e8c6262be50e5f2f93ae"
Oct 30 22:59:39.559844 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:59:39.559827    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07d76c93b87ca33785f521773aae7ffd6c450e56088e8c6262be50e5f2f93ae\": container with ID starting with d07d76c93b87ca33785f521773aae7ffd6c450e56088e8c6262be50e5f2f93ae not found: ID does not exist" containerID="d07d76c93b87ca33785f521773aae7ffd6c450e56088e8c6262be50e5f2f93ae"
Oct 30 22:59:39.559885 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.559850    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07d76c93b87ca33785f521773aae7ffd6c450e56088e8c6262be50e5f2f93ae"} err="failed to get container status \"d07d76c93b87ca33785f521773aae7ffd6c450e56088e8c6262be50e5f2f93ae\": rpc error: code = NotFound desc = could not find container \"d07d76c93b87ca33785f521773aae7ffd6c450e56088e8c6262be50e5f2f93ae\": container with ID starting with d07d76c93b87ca33785f521773aae7ffd6c450e56088e8c6262be50e5f2f93ae not found: ID does not exist"
Oct 30 22:59:39.703470 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.703435    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"]
Oct 30 22:59:39.709141 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:39.709117    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-stop-predictor-c7bb5549b-6v22x"]
Oct 30 22:59:40.158060 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:40.158028    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" path="/var/lib/kubelet/pods/20f94049-b2d3-4263-b790-f332bd7c1db6/volumes"
Oct 30 22:59:45.454994 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:45.454953    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 22:59:47.712917 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:47.712878    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"]
Oct 30 22:59:47.713493 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:47.713166    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kserve-container" containerID="cri-o://0eac5917fe06ba0eb858a93772c27ceaa557456440f8adbc5f56534bd88cd2da" gracePeriod=30
Oct 30 22:59:47.713493 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:47.713250    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kube-rbac-proxy" containerID="cri-o://787eba837e8775675559b8fdf7e0edf3a9f89f857d0b6cf43837b22db2cab440" gracePeriod=30
Oct 30 22:59:48.575681 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:48.575646    4947 generic.go:356] "Generic (PLEG): container finished" podID="f14a9890-8797-4f73-bd89-90c9235642a7" containerID="787eba837e8775675559b8fdf7e0edf3a9f89f857d0b6cf43837b22db2cab440" exitCode=0
Oct 30 22:59:48.575852 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:48.575718    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" event={"ID":"f14a9890-8797-4f73-bd89-90c9235642a7","Type":"ContainerDied","Data":"787eba837e8775675559b8fdf7e0edf3a9f89f857d0b6cf43837b22db2cab440"}
Oct 30 22:59:51.166045 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:51.166004    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.69:8643/healthz\": dial tcp 10.132.0.69:8643: connect: connection refused"
Oct 30 22:59:52.459545 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.459518    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:59:52.476696 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.476672    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f14a9890-8797-4f73-bd89-90c9235642a7-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f14a9890-8797-4f73-bd89-90c9235642a7-cabundle-cert\") pod \"f14a9890-8797-4f73-bd89-90c9235642a7\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") "
Oct 30 22:59:52.476799 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.476716    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14a9890-8797-4f73-bd89-90c9235642a7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f14a9890-8797-4f73-bd89-90c9235642a7-kserve-provision-location\") pod \"f14a9890-8797-4f73-bd89-90c9235642a7\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") "
Oct 30 22:59:52.476799 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.476748    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f14a9890-8797-4f73-bd89-90c9235642a7-kube-api-access-24vq6\" (UniqueName: \"kubernetes.io/projected/f14a9890-8797-4f73-bd89-90c9235642a7-kube-api-access-24vq6\") pod \"f14a9890-8797-4f73-bd89-90c9235642a7\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") "
Oct 30 22:59:52.476908 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.476799    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f14a9890-8797-4f73-bd89-90c9235642a7-proxy-tls\" (UniqueName: \"kubernetes.io/secret/f14a9890-8797-4f73-bd89-90c9235642a7-proxy-tls\") pod \"f14a9890-8797-4f73-bd89-90c9235642a7\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") "
Oct 30 22:59:52.476908 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.476823    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f14a9890-8797-4f73-bd89-90c9235642a7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f14a9890-8797-4f73-bd89-90c9235642a7-kube-rbac-proxy-sar-config\") pod \"f14a9890-8797-4f73-bd89-90c9235642a7\" (UID: \"f14a9890-8797-4f73-bd89-90c9235642a7\") "
Oct 30 22:59:52.477076 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.477045    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14a9890-8797-4f73-bd89-90c9235642a7-kserve-provision-location" pod "f14a9890-8797-4f73-bd89-90c9235642a7" (UID: "f14a9890-8797-4f73-bd89-90c9235642a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:59:52.477141 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.477082    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14a9890-8797-4f73-bd89-90c9235642a7-cabundle-cert" pod "f14a9890-8797-4f73-bd89-90c9235642a7" (UID: "f14a9890-8797-4f73-bd89-90c9235642a7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:59:52.477230 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.477210    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14a9890-8797-4f73-bd89-90c9235642a7-kube-rbac-proxy-sar-config" pod "f14a9890-8797-4f73-bd89-90c9235642a7" (UID: "f14a9890-8797-4f73-bd89-90c9235642a7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:59:52.479222 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.479197    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f14a9890-8797-4f73-bd89-90c9235642a7-proxy-tls" pod "f14a9890-8797-4f73-bd89-90c9235642a7" (UID: "f14a9890-8797-4f73-bd89-90c9235642a7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:59:52.479222 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.479207    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14a9890-8797-4f73-bd89-90c9235642a7-kube-api-access-24vq6" pod "f14a9890-8797-4f73-bd89-90c9235642a7" (UID: "f14a9890-8797-4f73-bd89-90c9235642a7"). InnerVolumeSpecName "kube-api-access-24vq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:59:52.577922 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.577843    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f14a9890-8797-4f73-bd89-90c9235642a7-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:59:52.577922 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.577870    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f14a9890-8797-4f73-bd89-90c9235642a7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:59:52.577922 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.577880    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f14a9890-8797-4f73-bd89-90c9235642a7-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:59:52.577922 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.577902    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f14a9890-8797-4f73-bd89-90c9235642a7-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:59:52.577922 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.577911    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24vq6\" (UniqueName: \"kubernetes.io/projected/f14a9890-8797-4f73-bd89-90c9235642a7-kube-api-access-24vq6\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:59:52.595185 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.595150    4947 generic.go:356] "Generic (PLEG): container finished" podID="f14a9890-8797-4f73-bd89-90c9235642a7" containerID="0eac5917fe06ba0eb858a93772c27ceaa557456440f8adbc5f56534bd88cd2da" exitCode=0
Oct 30 22:59:52.595352 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.595202    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" event={"ID":"f14a9890-8797-4f73-bd89-90c9235642a7","Type":"ContainerDied","Data":"0eac5917fe06ba0eb858a93772c27ceaa557456440f8adbc5f56534bd88cd2da"}
Oct 30 22:59:52.595352 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.595235    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"
Oct 30 22:59:52.595352 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.595298    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr" event={"ID":"f14a9890-8797-4f73-bd89-90c9235642a7","Type":"ContainerDied","Data":"beb59c80ca657b3293a034cb6d873fbf7236d46ed2ffeb1e5a53e22f46f2d63b"}
Oct 30 22:59:52.595352 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.595324    4947 scope.go:122] "RemoveContainer" containerID="787eba837e8775675559b8fdf7e0edf3a9f89f857d0b6cf43837b22db2cab440"
Oct 30 22:59:52.604266 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.604247    4947 scope.go:122] "RemoveContainer" containerID="0eac5917fe06ba0eb858a93772c27ceaa557456440f8adbc5f56534bd88cd2da"
Oct 30 22:59:52.612094 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.612076    4947 scope.go:122] "RemoveContainer" containerID="6e3b854a57a8e64a5cb40770b31bbd7aac480e3f01b0d2d9d633a467917c7a8d"
Oct 30 22:59:52.619243 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.619225    4947 scope.go:122] "RemoveContainer" containerID="787eba837e8775675559b8fdf7e0edf3a9f89f857d0b6cf43837b22db2cab440"
Oct 30 22:59:52.619532 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:59:52.619513    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"787eba837e8775675559b8fdf7e0edf3a9f89f857d0b6cf43837b22db2cab440\": container with ID starting with 787eba837e8775675559b8fdf7e0edf3a9f89f857d0b6cf43837b22db2cab440 not found: ID does not exist" containerID="787eba837e8775675559b8fdf7e0edf3a9f89f857d0b6cf43837b22db2cab440"
Oct 30 22:59:52.619592 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.619543    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787eba837e8775675559b8fdf7e0edf3a9f89f857d0b6cf43837b22db2cab440"} err="failed to get container status \"787eba837e8775675559b8fdf7e0edf3a9f89f857d0b6cf43837b22db2cab440\": rpc error: code = NotFound desc = could not find container \"787eba837e8775675559b8fdf7e0edf3a9f89f857d0b6cf43837b22db2cab440\": container with ID starting with 787eba837e8775675559b8fdf7e0edf3a9f89f857d0b6cf43837b22db2cab440 not found: ID does not exist"
Oct 30 22:59:52.619592 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.619561    4947 scope.go:122] "RemoveContainer" containerID="0eac5917fe06ba0eb858a93772c27ceaa557456440f8adbc5f56534bd88cd2da"
Oct 30 22:59:52.619798 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:59:52.619779    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eac5917fe06ba0eb858a93772c27ceaa557456440f8adbc5f56534bd88cd2da\": container with ID starting with 0eac5917fe06ba0eb858a93772c27ceaa557456440f8adbc5f56534bd88cd2da not found: ID does not exist" containerID="0eac5917fe06ba0eb858a93772c27ceaa557456440f8adbc5f56534bd88cd2da"
Oct 30 22:59:52.619857 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.619808    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eac5917fe06ba0eb858a93772c27ceaa557456440f8adbc5f56534bd88cd2da"} err="failed to get container status \"0eac5917fe06ba0eb858a93772c27ceaa557456440f8adbc5f56534bd88cd2da\": rpc error: code = NotFound desc = could not find container \"0eac5917fe06ba0eb858a93772c27ceaa557456440f8adbc5f56534bd88cd2da\": container with ID starting with 0eac5917fe06ba0eb858a93772c27ceaa557456440f8adbc5f56534bd88cd2da not found: ID does not exist"
Oct 30 22:59:52.619857 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.619832    4947 scope.go:122] "RemoveContainer" containerID="6e3b854a57a8e64a5cb40770b31bbd7aac480e3f01b0d2d9d633a467917c7a8d"
Oct 30 22:59:52.620079 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:59:52.620063    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e3b854a57a8e64a5cb40770b31bbd7aac480e3f01b0d2d9d633a467917c7a8d\": container with ID starting with 6e3b854a57a8e64a5cb40770b31bbd7aac480e3f01b0d2d9d633a467917c7a8d not found: ID does not exist" containerID="6e3b854a57a8e64a5cb40770b31bbd7aac480e3f01b0d2d9d633a467917c7a8d"
Oct 30 22:59:52.620139 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.620086    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3b854a57a8e64a5cb40770b31bbd7aac480e3f01b0d2d9d633a467917c7a8d"} err="failed to get container status \"6e3b854a57a8e64a5cb40770b31bbd7aac480e3f01b0d2d9d633a467917c7a8d\": rpc error: code = NotFound desc = could not find container \"6e3b854a57a8e64a5cb40770b31bbd7aac480e3f01b0d2d9d633a467917c7a8d\": container with ID starting with 6e3b854a57a8e64a5cb40770b31bbd7aac480e3f01b0d2d9d633a467917c7a8d not found: ID does not exist"
Oct 30 22:59:52.746356 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.746304    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"]
Oct 30 22:59:52.756588 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:52.756561    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-canary-stop-v2-predictor-75759d648c-tdmkr"]
Oct 30 22:59:54.159110 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:54.159078    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" path="/var/lib/kubelet/pods/f14a9890-8797-4f73-bd89-90c9235642a7/volumes"
Oct 30 22:59:55.455404 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:55.455364    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 22:59:56.464978 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.464946    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"]
Oct 30 22:59:56.465372 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.465250    4947 cpu_manager.go:414] "containerMap: removing container" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kube-rbac-proxy"
Oct 30 22:59:56.465372 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.465260    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kube-rbac-proxy"
Oct 30 22:59:56.465372 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.465279    4947 cpu_manager.go:414] "containerMap: removing container" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kserve-container"
Oct 30 22:59:56.465372 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.465285    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kserve-container"
Oct 30 22:59:56.465372 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.465302    4947 cpu_manager.go:414] "containerMap: removing container" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="storage-initializer"
Oct 30 22:59:56.465372 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.465307    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="storage-initializer"
Oct 30 22:59:56.465372 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.465318    4947 cpu_manager.go:414] "containerMap: removing container" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kube-rbac-proxy"
Oct 30 22:59:56.465372 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.465323    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kube-rbac-proxy"
Oct 30 22:59:56.465372 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.465345    4947 cpu_manager.go:414] "containerMap: removing container" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="storage-initializer"
Oct 30 22:59:56.465372 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.465350    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="storage-initializer"
Oct 30 22:59:56.465372 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.465357    4947 cpu_manager.go:414] "containerMap: removing container" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kserve-container"
Oct 30 22:59:56.465372 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.465362    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kserve-container"
Oct 30 22:59:56.465744 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.465409    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kube-rbac-proxy"
Oct 30 22:59:56.465744 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.465417    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kserve-container"
Oct 30 22:59:56.465744 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.465423    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f94049-b2d3-4263-b790-f332bd7c1db6" containerName="kube-rbac-proxy"
Oct 30 22:59:56.465744 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.465430    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f14a9890-8797-4f73-bd89-90c9235642a7" containerName="kserve-container"
Oct 30 22:59:56.468752 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.468737    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 22:59:56.508044 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.508015    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhk4p\" (UniqueName: \"kubernetes.io/projected/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kube-api-access-nhk4p\") pod \"isvc-lightgbm-predictor-88bd6d668-bhm8h\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 22:59:56.508189 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.508060    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-88bd6d668-bhm8h\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 22:59:56.508189 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.508081    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kserve-provision-location\") pod \"isvc-lightgbm-predictor-88bd6d668-bhm8h\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 22:59:56.508304 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.508215    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-proxy-tls\") pod \"isvc-lightgbm-predictor-88bd6d668-bhm8h\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 22:59:56.508360 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.508313    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-cabundle-cert\") pod \"isvc-lightgbm-predictor-88bd6d668-bhm8h\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 22:59:56.522840 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.522816    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"seaweedfs-s3-creds\""
Oct 30 22:59:56.522956 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.522947    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Oct 30 22:59:56.572160 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.572131    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"kube-root-ca.crt\""
Oct 30 22:59:56.572292 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.572222    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"openshift-service-ca.crt\""
Oct 30 22:59:56.572292 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.572232    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"default-dockercfg-s9cqw\""
Oct 30 22:59:56.572292 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.572249    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:59:56.572292 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.572261    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-lightgbm-predictor-serving-cert\""
Oct 30 22:59:56.573839 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.573822    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"]
Oct 30 22:59:56.608832 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.608803    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhk4p\" (UniqueName: \"kubernetes.io/projected/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kube-api-access-nhk4p\") pod \"isvc-lightgbm-predictor-88bd6d668-bhm8h\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 22:59:56.608976 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.608865    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-88bd6d668-bhm8h\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 22:59:56.608976 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.608886    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kserve-provision-location\") pod \"isvc-lightgbm-predictor-88bd6d668-bhm8h\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 22:59:56.608976 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.608908    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-proxy-tls\") pod \"isvc-lightgbm-predictor-88bd6d668-bhm8h\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 22:59:56.609132 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.609021    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-cabundle-cert\") pod \"isvc-lightgbm-predictor-88bd6d668-bhm8h\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 22:59:56.609406 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.609383    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kserve-provision-location\") pod \"isvc-lightgbm-predictor-88bd6d668-bhm8h\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 22:59:56.609714 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.609695    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-88bd6d668-bhm8h\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 22:59:56.609790 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.609727    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-cabundle-cert\") pod \"isvc-lightgbm-predictor-88bd6d668-bhm8h\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 22:59:56.611957 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.611933    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-proxy-tls\") pod \"isvc-lightgbm-predictor-88bd6d668-bhm8h\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 22:59:56.671234 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.671210    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhk4p\" (UniqueName: \"kubernetes.io/projected/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kube-api-access-nhk4p\") pod \"isvc-lightgbm-predictor-88bd6d668-bhm8h\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 22:59:56.784883 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:56.784801    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 22:59:57.209616 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:57.209543    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"]
Oct 30 22:59:57.212531 ip-10-0-133-218 kubenswrapper[4947]: W1030 22:59:57.212507    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3adfa4eb_16b1_4cfd_b34b_7029e32404e2.slice/crio-0a57f4aef276cb3f02182b227a8cc0626f7bfbea66d5822ca132b3333ad77082 WatchSource:0}: Error finding container 0a57f4aef276cb3f02182b227a8cc0626f7bfbea66d5822ca132b3333ad77082: Status 404 returned error can't find the container with id 0a57f4aef276cb3f02182b227a8cc0626f7bfbea66d5822ca132b3333ad77082
Oct 30 22:59:57.620960 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:57.620925    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" event={"ID":"3adfa4eb-16b1-4cfd-b34b-7029e32404e2","Type":"ContainerStarted","Data":"6448de7df2e7915901f977e1ec3c3d84b7ab6f4193dd524cba748119713515a6"}
Oct 30 22:59:57.621472 ip-10-0-133-218 kubenswrapper[4947]: I1030 22:59:57.621002    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" event={"ID":"3adfa4eb-16b1-4cfd-b34b-7029e32404e2","Type":"ContainerStarted","Data":"0a57f4aef276cb3f02182b227a8cc0626f7bfbea66d5822ca132b3333ad77082"}
Oct 30 22:59:59.047979 ip-10-0-133-218 kubenswrapper[4947]: E1030 22:59:59.047933    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:00:01.638721 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:01.638688    4947 generic.go:356] "Generic (PLEG): container finished" podID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerID="6448de7df2e7915901f977e1ec3c3d84b7ab6f4193dd524cba748119713515a6" exitCode=0
Oct 30 23:00:01.639147 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:01.638758    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" event={"ID":"3adfa4eb-16b1-4cfd-b34b-7029e32404e2","Type":"ContainerDied","Data":"6448de7df2e7915901f977e1ec3c3d84b7ab6f4193dd524cba748119713515a6"}
Oct 30 23:00:01.640000 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:01.639984    4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:00:05.454660 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:05.454614    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 23:00:10.689216 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:10.689139    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" event={"ID":"3adfa4eb-16b1-4cfd-b34b-7029e32404e2","Type":"ContainerStarted","Data":"ac86d59bb54c8d1c37c6aa47f1b17623825d78e768fdf32dbb18444905213545"}
Oct 30 23:00:10.689216 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:10.689197    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" event={"ID":"3adfa4eb-16b1-4cfd-b34b-7029e32404e2","Type":"ContainerStarted","Data":"7a92b86910acfe070a37c57ec68c58122f330a30871c60daa0bce3a7ef4c6f8e"}
Oct 30 23:00:10.689640 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:10.689383    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 23:00:10.860697 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:10.860630    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" podStartSLOduration=2.324205402 podStartE2EDuration="14.860616328s" totalImagesPullingTime="8.652809876s" totalInitContainerRuntime="3.88360105s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:59:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:10.860111977 +0000 UTC m=+3108.859459542" watchObservedRunningTime="2026-10-30 23:00:10.860616328 +0000 UTC m=+3108.859963936"
Oct 30 23:00:11.697087 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:11.697053    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 23:00:11.698729 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:11.698701    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 23:00:12.702741 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:12.702696    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 23:00:15.454515 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:15.454472    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 23:00:17.707092 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:17.707065    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 23:00:17.707541 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:17.707513    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 23:00:22.623453 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:22.623416    4947 scope.go:122] "RemoveContainer" containerID="a90a0f6fde263641ce362bdb547ea2fc637b6207ba7a050e386928e4515366f1"
Oct 30 23:00:22.631437 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:22.631418    4947 scope.go:122] "RemoveContainer" containerID="e36c09b90d191a9ec59e743ddafa3b0204763d7804d55fc901618b34ce179c09"
Oct 30 23:00:25.454945 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:25.454851    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 23:00:27.707547 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:27.707507    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 23:00:29.048220 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:00:29.048180    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:00:35.454558 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:35.454513    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 23:00:37.707550 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:37.707511    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 23:00:43.154457 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:43.154427    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 23:00:46.400814 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:46.400777    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"]
Oct 30 23:00:46.401295 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:46.401098    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kserve-container" containerID="cri-o://a8358dcedd7ca0c88ec21d8c5dc0ee07be9824d88fd7aba0cfbb6f5557d755b4" gracePeriod=30
Oct 30 23:00:46.401295 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:46.401116    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kube-rbac-proxy" containerID="cri-o://e4e46bbffaf6b611d112f56a81b23c72288d2b5c71190a62f0f2946bfbeecc4b" gracePeriod=30
Oct 30 23:00:46.844132 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:46.844100    4947 generic.go:356] "Generic (PLEG): container finished" podID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerID="e4e46bbffaf6b611d112f56a81b23c72288d2b5c71190a62f0f2946bfbeecc4b" exitCode=0
Oct 30 23:00:46.844303 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:46.844143    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" event={"ID":"8284975a-02bf-4d35-b6ea-28edaa42e622","Type":"ContainerDied","Data":"e4e46bbffaf6b611d112f56a81b23c72288d2b5c71190a62f0f2946bfbeecc4b"}
Oct 30 23:00:47.707879 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:47.707844    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 23:00:50.450190 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:50.450143    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.71:8643/healthz\": dial tcp 10.132.0.71:8643: connect: connection refused"
Oct 30 23:00:51.647525 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.647499    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 23:00:51.669938 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.669906    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8284975a-02bf-4d35-b6ea-28edaa42e622-proxy-tls\" (UniqueName: \"kubernetes.io/secret/8284975a-02bf-4d35-b6ea-28edaa42e622-proxy-tls\") pod \"8284975a-02bf-4d35-b6ea-28edaa42e622\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") "
Oct 30 23:00:51.670077 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.669956    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8284975a-02bf-4d35-b6ea-28edaa42e622-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8284975a-02bf-4d35-b6ea-28edaa42e622-kserve-provision-location\") pod \"8284975a-02bf-4d35-b6ea-28edaa42e622\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") "
Oct 30 23:00:51.670077 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.669998    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8284975a-02bf-4d35-b6ea-28edaa42e622-kube-api-access-g6xx4\" (UniqueName: \"kubernetes.io/projected/8284975a-02bf-4d35-b6ea-28edaa42e622-kube-api-access-g6xx4\") pod \"8284975a-02bf-4d35-b6ea-28edaa42e622\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") "
Oct 30 23:00:51.670077 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.670016    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8284975a-02bf-4d35-b6ea-28edaa42e622-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8284975a-02bf-4d35-b6ea-28edaa42e622-kube-rbac-proxy-sar-config\") pod \"8284975a-02bf-4d35-b6ea-28edaa42e622\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") "
Oct 30 23:00:51.670077 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.670046    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8284975a-02bf-4d35-b6ea-28edaa42e622-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8284975a-02bf-4d35-b6ea-28edaa42e622-cabundle-cert\") pod \"8284975a-02bf-4d35-b6ea-28edaa42e622\" (UID: \"8284975a-02bf-4d35-b6ea-28edaa42e622\") "
Oct 30 23:00:51.670353 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.670300    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8284975a-02bf-4d35-b6ea-28edaa42e622-kserve-provision-location" pod "8284975a-02bf-4d35-b6ea-28edaa42e622" (UID: "8284975a-02bf-4d35-b6ea-28edaa42e622"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:00:51.670495 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.670428    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8284975a-02bf-4d35-b6ea-28edaa42e622-kube-rbac-proxy-sar-config" pod "8284975a-02bf-4d35-b6ea-28edaa42e622" (UID: "8284975a-02bf-4d35-b6ea-28edaa42e622"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:51.670495 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.670486    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8284975a-02bf-4d35-b6ea-28edaa42e622-cabundle-cert" pod "8284975a-02bf-4d35-b6ea-28edaa42e622" (UID: "8284975a-02bf-4d35-b6ea-28edaa42e622"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:51.672256 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.672235    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8284975a-02bf-4d35-b6ea-28edaa42e622-proxy-tls" pod "8284975a-02bf-4d35-b6ea-28edaa42e622" (UID: "8284975a-02bf-4d35-b6ea-28edaa42e622"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:00:51.672368 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.672352    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8284975a-02bf-4d35-b6ea-28edaa42e622-kube-api-access-g6xx4" pod "8284975a-02bf-4d35-b6ea-28edaa42e622" (UID: "8284975a-02bf-4d35-b6ea-28edaa42e622"). InnerVolumeSpecName "kube-api-access-g6xx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:51.771601 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.771509    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8284975a-02bf-4d35-b6ea-28edaa42e622-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:00:51.771601 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.771542    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8284975a-02bf-4d35-b6ea-28edaa42e622-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:00:51.771601 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.771555    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6xx4\" (UniqueName: \"kubernetes.io/projected/8284975a-02bf-4d35-b6ea-28edaa42e622-kube-api-access-g6xx4\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:00:51.771601 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.771569    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8284975a-02bf-4d35-b6ea-28edaa42e622-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:00:51.771601 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.771581    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8284975a-02bf-4d35-b6ea-28edaa42e622-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:00:51.867890 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.867850    4947 generic.go:356] "Generic (PLEG): container finished" podID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerID="a8358dcedd7ca0c88ec21d8c5dc0ee07be9824d88fd7aba0cfbb6f5557d755b4" exitCode=0
Oct 30 23:00:51.868068 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.867915    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"
Oct 30 23:00:51.868068 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.867911    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" event={"ID":"8284975a-02bf-4d35-b6ea-28edaa42e622","Type":"ContainerDied","Data":"a8358dcedd7ca0c88ec21d8c5dc0ee07be9824d88fd7aba0cfbb6f5557d755b4"}
Oct 30 23:00:51.868068 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.867980    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr" event={"ID":"8284975a-02bf-4d35-b6ea-28edaa42e622","Type":"ContainerDied","Data":"dd7e5895e434740770df5ccd1b02c36df4cfe4133125ce91da47f715e3a72701"}
Oct 30 23:00:51.868068 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.868010    4947 scope.go:122] "RemoveContainer" containerID="e4e46bbffaf6b611d112f56a81b23c72288d2b5c71190a62f0f2946bfbeecc4b"
Oct 30 23:00:51.883363 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.883315    4947 scope.go:122] "RemoveContainer" containerID="a8358dcedd7ca0c88ec21d8c5dc0ee07be9824d88fd7aba0cfbb6f5557d755b4"
Oct 30 23:00:51.891121 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.891103    4947 scope.go:122] "RemoveContainer" containerID="143329f5f41a5acfcfdd9eb9b1d5148588f6cc14d34f988741c97d0c9f650e1d"
Oct 30 23:00:51.898649 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.898628    4947 scope.go:122] "RemoveContainer" containerID="e4e46bbffaf6b611d112f56a81b23c72288d2b5c71190a62f0f2946bfbeecc4b"
Oct 30 23:00:51.898952 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:00:51.898922    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e46bbffaf6b611d112f56a81b23c72288d2b5c71190a62f0f2946bfbeecc4b\": container with ID starting with e4e46bbffaf6b611d112f56a81b23c72288d2b5c71190a62f0f2946bfbeecc4b not found: ID does not exist" containerID="e4e46bbffaf6b611d112f56a81b23c72288d2b5c71190a62f0f2946bfbeecc4b"
Oct 30 23:00:51.898999 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.898965    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e46bbffaf6b611d112f56a81b23c72288d2b5c71190a62f0f2946bfbeecc4b"} err="failed to get container status \"e4e46bbffaf6b611d112f56a81b23c72288d2b5c71190a62f0f2946bfbeecc4b\": rpc error: code = NotFound desc = could not find container \"e4e46bbffaf6b611d112f56a81b23c72288d2b5c71190a62f0f2946bfbeecc4b\": container with ID starting with e4e46bbffaf6b611d112f56a81b23c72288d2b5c71190a62f0f2946bfbeecc4b not found: ID does not exist"
Oct 30 23:00:51.898999 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.898991    4947 scope.go:122] "RemoveContainer" containerID="a8358dcedd7ca0c88ec21d8c5dc0ee07be9824d88fd7aba0cfbb6f5557d755b4"
Oct 30 23:00:51.899258 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:00:51.899240    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8358dcedd7ca0c88ec21d8c5dc0ee07be9824d88fd7aba0cfbb6f5557d755b4\": container with ID starting with a8358dcedd7ca0c88ec21d8c5dc0ee07be9824d88fd7aba0cfbb6f5557d755b4 not found: ID does not exist" containerID="a8358dcedd7ca0c88ec21d8c5dc0ee07be9824d88fd7aba0cfbb6f5557d755b4"
Oct 30 23:00:51.899315 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.899264    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8358dcedd7ca0c88ec21d8c5dc0ee07be9824d88fd7aba0cfbb6f5557d755b4"} err="failed to get container status \"a8358dcedd7ca0c88ec21d8c5dc0ee07be9824d88fd7aba0cfbb6f5557d755b4\": rpc error: code = NotFound desc = could not find container \"a8358dcedd7ca0c88ec21d8c5dc0ee07be9824d88fd7aba0cfbb6f5557d755b4\": container with ID starting with a8358dcedd7ca0c88ec21d8c5dc0ee07be9824d88fd7aba0cfbb6f5557d755b4 not found: ID does not exist"
Oct 30 23:00:51.899315 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.899279    4947 scope.go:122] "RemoveContainer" containerID="143329f5f41a5acfcfdd9eb9b1d5148588f6cc14d34f988741c97d0c9f650e1d"
Oct 30 23:00:51.899576 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:00:51.899558    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143329f5f41a5acfcfdd9eb9b1d5148588f6cc14d34f988741c97d0c9f650e1d\": container with ID starting with 143329f5f41a5acfcfdd9eb9b1d5148588f6cc14d34f988741c97d0c9f650e1d not found: ID does not exist" containerID="143329f5f41a5acfcfdd9eb9b1d5148588f6cc14d34f988741c97d0c9f650e1d"
Oct 30 23:00:51.899626 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:51.899582    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143329f5f41a5acfcfdd9eb9b1d5148588f6cc14d34f988741c97d0c9f650e1d"} err="failed to get container status \"143329f5f41a5acfcfdd9eb9b1d5148588f6cc14d34f988741c97d0c9f650e1d\": rpc error: code = NotFound desc = could not find container \"143329f5f41a5acfcfdd9eb9b1d5148588f6cc14d34f988741c97d0c9f650e1d\": container with ID starting with 143329f5f41a5acfcfdd9eb9b1d5148588f6cc14d34f988741c97d0c9f650e1d not found: ID does not exist"
Oct 30 23:00:52.035749 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:52.035719    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"]
Oct 30 23:00:52.042916 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:52.042887    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-custom-pass-predictor-5844bfbccd-dbfbr"]
Oct 30 23:00:52.158842 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:52.158809    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" path="/var/lib/kubelet/pods/8284975a-02bf-4d35-b6ea-28edaa42e622/volumes"
Oct 30 23:00:57.708433 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:00:57.708387    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 23:00:59.048477 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:00:59.048440    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:01:03.812883 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.812853    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"]
Oct 30 23:01:03.813281 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.813196    4947 cpu_manager.go:414] "containerMap: removing container" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kube-rbac-proxy"
Oct 30 23:01:03.813281 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.813210    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kube-rbac-proxy"
Oct 30 23:01:03.813281 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.813224    4947 cpu_manager.go:414] "containerMap: removing container" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kserve-container"
Oct 30 23:01:03.813281 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.813229    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kserve-container"
Oct 30 23:01:03.813281 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.813254    4947 cpu_manager.go:414] "containerMap: removing container" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="storage-initializer"
Oct 30 23:01:03.813281 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.813260    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="storage-initializer"
Oct 30 23:01:03.813500 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.813322    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kube-rbac-proxy"
Oct 30 23:01:03.813500 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.813347    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="8284975a-02bf-4d35-b6ea-28edaa42e622" containerName="kserve-container"
Oct 30 23:01:03.815407 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.815390    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:03.869208 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.869173    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Oct 30 23:01:03.875575 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.875545    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22a16a9c-f385-419b-938e-0732450d4903-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:03.875691 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.875592    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a16a9c-f385-419b-938e-0732450d4903-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:03.875691 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.875660    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22a16a9c-f385-419b-938e-0732450d4903-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:03.875780 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.875700    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22a16a9c-f385-419b-938e-0732450d4903-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:03.875823 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.875782    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlkhs\" (UniqueName: \"kubernetes.io/projected/22a16a9c-f385-419b-938e-0732450d4903-kube-api-access-mlkhs\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:03.918680 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.918544    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 23:01:03.918680 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.918600    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 23:01:03.919058 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.918813    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 23:01:03.919058 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.918904    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"storage-config\""
Oct 30 23:01:03.919058 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.918915    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Oct 30 23:01:03.919058 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.918968    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 23:01:03.920581 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.920557    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"]
Oct 30 23:01:03.976910 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.976867    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlkhs\" (UniqueName: \"kubernetes.io/projected/22a16a9c-f385-419b-938e-0732450d4903-kube-api-access-mlkhs\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:03.977083 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.976938    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22a16a9c-f385-419b-938e-0732450d4903-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:03.977083 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.976963    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a16a9c-f385-419b-938e-0732450d4903-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:03.977083 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.976991    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22a16a9c-f385-419b-938e-0732450d4903-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:03.977083 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.977014    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22a16a9c-f385-419b-938e-0732450d4903-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:03.977468 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.977445    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a16a9c-f385-419b-938e-0732450d4903-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:03.977801 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.977782    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22a16a9c-f385-419b-938e-0732450d4903-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:03.977855 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.977786    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22a16a9c-f385-419b-938e-0732450d4903-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:03.979598 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:03.979580    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22a16a9c-f385-419b-938e-0732450d4903-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:04.040155 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:04.040115    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlkhs\" (UniqueName: \"kubernetes.io/projected/22a16a9c-f385-419b-938e-0732450d4903-kube-api-access-mlkhs\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") " pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:04.129743 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:04.129656    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:04.561864 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:04.557552    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"]
Oct 30 23:01:04.565255 ip-10-0-133-218 kubenswrapper[4947]: W1030 23:01:04.565220    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a16a9c_f385_419b_938e_0732450d4903.slice/crio-a524e37aadd30cc24e2b945fa5c12bb032047a66762af3b02e667dcad0393794 WatchSource:0}: Error finding container a524e37aadd30cc24e2b945fa5c12bb032047a66762af3b02e667dcad0393794: Status 404 returned error can't find the container with id a524e37aadd30cc24e2b945fa5c12bb032047a66762af3b02e667dcad0393794
Oct 30 23:01:04.927850 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:04.927746    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" event={"ID":"22a16a9c-f385-419b-938e-0732450d4903","Type":"ContainerStarted","Data":"ca08996891db8f9129bf5593255cc7bf5723c7e68dab30c40a6b90f751352d92"}
Oct 30 23:01:04.927850 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:04.927824    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" event={"ID":"22a16a9c-f385-419b-938e-0732450d4903","Type":"ContainerStarted","Data":"a524e37aadd30cc24e2b945fa5c12bb032047a66762af3b02e667dcad0393794"}
Oct 30 23:01:05.933795 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:05.933758    4947 generic.go:356] "Generic (PLEG): container finished" podID="22a16a9c-f385-419b-938e-0732450d4903" containerID="ca08996891db8f9129bf5593255cc7bf5723c7e68dab30c40a6b90f751352d92" exitCode=0
Oct 30 23:01:05.934187 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:05.933811    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" event={"ID":"22a16a9c-f385-419b-938e-0732450d4903","Type":"ContainerDied","Data":"ca08996891db8f9129bf5593255cc7bf5723c7e68dab30c40a6b90f751352d92"}
Oct 30 23:01:06.939982 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:06.939945    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" event={"ID":"22a16a9c-f385-419b-938e-0732450d4903","Type":"ContainerStarted","Data":"ae2a0d86cebaa7282201b74fbd89601e71ccfae4f6f82840d0f232e6aed4001f"}
Oct 30 23:01:06.940387 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:06.940010    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" event={"ID":"22a16a9c-f385-419b-938e-0732450d4903","Type":"ContainerStarted","Data":"7b2a72cdf5be97a40ef8bb68a53360e9003a51b505147bd90b6b2d532caa0e8a"}
Oct 30 23:01:06.940387 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:06.940146    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:07.062189 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:07.062126    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" podStartSLOduration=2.9500291819999998 podStartE2EDuration="4.062104585s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.112075403s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:01:07.061801071 +0000 UTC m=+3165.061148636" watchObservedRunningTime="2026-10-30 23:01:07.062104585 +0000 UTC m=+3165.061452150"
Oct 30 23:01:07.708132 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:07.708097    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 23:01:07.944741 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:07.944709    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:07.946216 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:07.946179    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 23:01:08.948968 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:08.948932    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 23:01:13.953464 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:13.953434    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:01:13.953908 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:13.953883    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 23:01:17.708398 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:17.708349    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 23:01:23.954428 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:23.954390    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 23:01:27.708407 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:27.708365    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 23:01:29.047674 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:01:29.047628    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:01:33.954903 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:33.954825    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 23:01:37.708202 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:37.708158    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 23:01:43.954591 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:43.954543    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 23:01:47.153513 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:47.153486    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 23:01:53.954454 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:53.954417    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 23:01:57.468762 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:57.468723    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"]
Oct 30 23:01:57.469262 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:57.468977    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kserve-container" containerID="cri-o://7a92b86910acfe070a37c57ec68c58122f330a30871c60daa0bce3a7ef4c6f8e" gracePeriod=30
Oct 30 23:01:57.469262 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:57.469018    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kube-rbac-proxy" containerID="cri-o://ac86d59bb54c8d1c37c6aa47f1b17623825d78e768fdf32dbb18444905213545" gracePeriod=30
Oct 30 23:01:57.702682 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:57.702640    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.72:8643/healthz\": dial tcp 10.132.0.72:8643: connect: connection refused"
Oct 30 23:01:58.157619 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:58.157587    4947 generic.go:356] "Generic (PLEG): container finished" podID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerID="ac86d59bb54c8d1c37c6aa47f1b17623825d78e768fdf32dbb18444905213545" exitCode=0
Oct 30 23:01:58.159219 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:01:58.159198    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" event={"ID":"3adfa4eb-16b1-4cfd-b34b-7029e32404e2","Type":"ContainerDied","Data":"ac86d59bb54c8d1c37c6aa47f1b17623825d78e768fdf32dbb18444905213545"}
Oct 30 23:01:59.048393 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:01:59.048353    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:02:02.702869 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:02.702825    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.72:8643/healthz\": dial tcp 10.132.0.72:8643: connect: connection refused"
Oct 30 23:02:02.909699 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:02.909672    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 23:02:02.978052 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:02.977976    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-proxy-tls\" (UniqueName: \"kubernetes.io/secret/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-proxy-tls\") pod \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") "
Oct 30 23:02:02.978188 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:02.978095    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kube-rbac-proxy-sar-config\") pod \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") "
Oct 30 23:02:02.978188 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:02.978129    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kserve-provision-location\") pod \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") "
Oct 30 23:02:02.978188 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:02.978159    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kube-api-access-nhk4p\" (UniqueName: \"kubernetes.io/projected/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kube-api-access-nhk4p\") pod \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") "
Oct 30 23:02:02.978373 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:02.978188    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-cabundle-cert\") pod \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\" (UID: \"3adfa4eb-16b1-4cfd-b34b-7029e32404e2\") "
Oct 30 23:02:02.978559 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:02.978518    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kserve-provision-location" pod "3adfa4eb-16b1-4cfd-b34b-7029e32404e2" (UID: "3adfa4eb-16b1-4cfd-b34b-7029e32404e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:02.978559 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:02.978523    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kube-rbac-proxy-sar-config" pod "3adfa4eb-16b1-4cfd-b34b-7029e32404e2" (UID: "3adfa4eb-16b1-4cfd-b34b-7029e32404e2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:02.978681 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:02.978578    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-cabundle-cert" pod "3adfa4eb-16b1-4cfd-b34b-7029e32404e2" (UID: "3adfa4eb-16b1-4cfd-b34b-7029e32404e2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:02.980257 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:02.980236    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-proxy-tls" pod "3adfa4eb-16b1-4cfd-b34b-7029e32404e2" (UID: "3adfa4eb-16b1-4cfd-b34b-7029e32404e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:02.980651 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:02.980633    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kube-api-access-nhk4p" pod "3adfa4eb-16b1-4cfd-b34b-7029e32404e2" (UID: "3adfa4eb-16b1-4cfd-b34b-7029e32404e2"). InnerVolumeSpecName "kube-api-access-nhk4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:03.079154 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.079126    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:02:03.079154 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.079149    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:02:03.079354 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.079162    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:02:03.079354 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.079172    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhk4p\" (UniqueName: \"kubernetes.io/projected/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-kube-api-access-nhk4p\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:02:03.079354 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.079181    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3adfa4eb-16b1-4cfd-b34b-7029e32404e2-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:02:03.177749 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.177715    4947 generic.go:356] "Generic (PLEG): container finished" podID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerID="7a92b86910acfe070a37c57ec68c58122f330a30871c60daa0bce3a7ef4c6f8e" exitCode=0
Oct 30 23:02:03.177926 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.177774    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"
Oct 30 23:02:03.177926 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.177789    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" event={"ID":"3adfa4eb-16b1-4cfd-b34b-7029e32404e2","Type":"ContainerDied","Data":"7a92b86910acfe070a37c57ec68c58122f330a30871c60daa0bce3a7ef4c6f8e"}
Oct 30 23:02:03.177926 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.177849    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h" event={"ID":"3adfa4eb-16b1-4cfd-b34b-7029e32404e2","Type":"ContainerDied","Data":"0a57f4aef276cb3f02182b227a8cc0626f7bfbea66d5822ca132b3333ad77082"}
Oct 30 23:02:03.177926 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.177877    4947 scope.go:122] "RemoveContainer" containerID="ac86d59bb54c8d1c37c6aa47f1b17623825d78e768fdf32dbb18444905213545"
Oct 30 23:02:03.187229 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.187213    4947 scope.go:122] "RemoveContainer" containerID="7a92b86910acfe070a37c57ec68c58122f330a30871c60daa0bce3a7ef4c6f8e"
Oct 30 23:02:03.194607 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.194591    4947 scope.go:122] "RemoveContainer" containerID="6448de7df2e7915901f977e1ec3c3d84b7ab6f4193dd524cba748119713515a6"
Oct 30 23:02:03.201433 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.201404    4947 scope.go:122] "RemoveContainer" containerID="ac86d59bb54c8d1c37c6aa47f1b17623825d78e768fdf32dbb18444905213545"
Oct 30 23:02:03.201690 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:02:03.201675    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac86d59bb54c8d1c37c6aa47f1b17623825d78e768fdf32dbb18444905213545\": container with ID starting with ac86d59bb54c8d1c37c6aa47f1b17623825d78e768fdf32dbb18444905213545 not found: ID does not exist" containerID="ac86d59bb54c8d1c37c6aa47f1b17623825d78e768fdf32dbb18444905213545"
Oct 30 23:02:03.201743 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.201699    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac86d59bb54c8d1c37c6aa47f1b17623825d78e768fdf32dbb18444905213545"} err="failed to get container status \"ac86d59bb54c8d1c37c6aa47f1b17623825d78e768fdf32dbb18444905213545\": rpc error: code = NotFound desc = could not find container \"ac86d59bb54c8d1c37c6aa47f1b17623825d78e768fdf32dbb18444905213545\": container with ID starting with ac86d59bb54c8d1c37c6aa47f1b17623825d78e768fdf32dbb18444905213545 not found: ID does not exist"
Oct 30 23:02:03.201743 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.201720    4947 scope.go:122] "RemoveContainer" containerID="7a92b86910acfe070a37c57ec68c58122f330a30871c60daa0bce3a7ef4c6f8e"
Oct 30 23:02:03.201949 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:02:03.201931    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a92b86910acfe070a37c57ec68c58122f330a30871c60daa0bce3a7ef4c6f8e\": container with ID starting with 7a92b86910acfe070a37c57ec68c58122f330a30871c60daa0bce3a7ef4c6f8e not found: ID does not exist" containerID="7a92b86910acfe070a37c57ec68c58122f330a30871c60daa0bce3a7ef4c6f8e"
Oct 30 23:02:03.201999 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.201957    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a92b86910acfe070a37c57ec68c58122f330a30871c60daa0bce3a7ef4c6f8e"} err="failed to get container status \"7a92b86910acfe070a37c57ec68c58122f330a30871c60daa0bce3a7ef4c6f8e\": rpc error: code = NotFound desc = could not find container \"7a92b86910acfe070a37c57ec68c58122f330a30871c60daa0bce3a7ef4c6f8e\": container with ID starting with 7a92b86910acfe070a37c57ec68c58122f330a30871c60daa0bce3a7ef4c6f8e not found: ID does not exist"
Oct 30 23:02:03.201999 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.201971    4947 scope.go:122] "RemoveContainer" containerID="6448de7df2e7915901f977e1ec3c3d84b7ab6f4193dd524cba748119713515a6"
Oct 30 23:02:03.202183 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:02:03.202168    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6448de7df2e7915901f977e1ec3c3d84b7ab6f4193dd524cba748119713515a6\": container with ID starting with 6448de7df2e7915901f977e1ec3c3d84b7ab6f4193dd524cba748119713515a6 not found: ID does not exist" containerID="6448de7df2e7915901f977e1ec3c3d84b7ab6f4193dd524cba748119713515a6"
Oct 30 23:02:03.202224 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.202185    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6448de7df2e7915901f977e1ec3c3d84b7ab6f4193dd524cba748119713515a6"} err="failed to get container status \"6448de7df2e7915901f977e1ec3c3d84b7ab6f4193dd524cba748119713515a6\": rpc error: code = NotFound desc = could not find container \"6448de7df2e7915901f977e1ec3c3d84b7ab6f4193dd524cba748119713515a6\": container with ID starting with 6448de7df2e7915901f977e1ec3c3d84b7ab6f4193dd524cba748119713515a6 not found: ID does not exist"
Oct 30 23:02:03.347842 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.347813    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"]
Oct 30 23:02:03.350671 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.350648    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-predictor-88bd6d668-bhm8h"]
Oct 30 23:02:03.954633 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:03.954599    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 23:02:04.158182 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:04.158147    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" path="/var/lib/kubelet/pods/3adfa4eb-16b1-4cfd-b34b-7029e32404e2/volumes"
Oct 30 23:02:06.567014 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.566986    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"]
Oct 30 23:02:06.567403 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.567293    4947 cpu_manager.go:414] "containerMap: removing container" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="storage-initializer"
Oct 30 23:02:06.567403 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.567304    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="storage-initializer"
Oct 30 23:02:06.567403 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.567315    4947 cpu_manager.go:414] "containerMap: removing container" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kube-rbac-proxy"
Oct 30 23:02:06.567403 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.567320    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kube-rbac-proxy"
Oct 30 23:02:06.567403 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.567365    4947 cpu_manager.go:414] "containerMap: removing container" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kserve-container"
Oct 30 23:02:06.567403 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.567372    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kserve-container"
Oct 30 23:02:06.567589 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.567435    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kube-rbac-proxy"
Oct 30 23:02:06.567589 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.567446    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="3adfa4eb-16b1-4cfd-b34b-7029e32404e2" containerName="kserve-container"
Oct 30 23:02:06.570563 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.570546    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:06.609160 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.609131    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwd48\" (UniqueName: \"kubernetes.io/projected/380609b9-e345-4539-a18e-b63024b9621e-kube-api-access-zwd48\") pod \"isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:06.609280 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.609163    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/380609b9-e345-4539-a18e-b63024b9621e-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:06.609280 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.609249    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/380609b9-e345-4539-a18e-b63024b9621e-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:06.609280 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.609270    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/380609b9-e345-4539-a18e-b63024b9621e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:06.609413 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.609317    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380609b9-e345-4539-a18e-b63024b9621e-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:06.673717 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.673671    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Oct 30 23:02:06.673717 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.673691    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"default-dockercfg-s9cqw\""
Oct 30 23:02:06.673717 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.673704    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"kube-root-ca.crt\""
Oct 30 23:02:06.673717 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.673712    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"seaweedfs-s3-creds\""
Oct 30 23:02:06.674019 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.673763    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 23:02:06.674019 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.673790    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"openshift-service-ca.crt\""
Oct 30 23:02:06.674019 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.673928    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Oct 30 23:02:06.675296 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.675272    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"]
Oct 30 23:02:06.710547 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.710521    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwd48\" (UniqueName: \"kubernetes.io/projected/380609b9-e345-4539-a18e-b63024b9621e-kube-api-access-zwd48\") pod \"isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:06.710659 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.710550    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/380609b9-e345-4539-a18e-b63024b9621e-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:06.710659 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.710582    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/380609b9-e345-4539-a18e-b63024b9621e-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:06.710744 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.710699    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/380609b9-e345-4539-a18e-b63024b9621e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:06.710790 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.710774    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380609b9-e345-4539-a18e-b63024b9621e-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:06.711088 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.711071    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380609b9-e345-4539-a18e-b63024b9621e-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:06.711236 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.711217    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/380609b9-e345-4539-a18e-b63024b9621e-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:06.711317 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.711300    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/380609b9-e345-4539-a18e-b63024b9621e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:06.713249 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.713231    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/380609b9-e345-4539-a18e-b63024b9621e-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:06.773615 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.773588    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwd48\" (UniqueName: \"kubernetes.io/projected/380609b9-e345-4539-a18e-b63024b9621e-kube-api-access-zwd48\") pod \"isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:06.884565 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:06.884483    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:07.309579 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:07.309549    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"]
Oct 30 23:02:07.312834 ip-10-0-133-218 kubenswrapper[4947]: W1030 23:02:07.312809    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380609b9_e345_4539_a18e_b63024b9621e.slice/crio-348f803f2c104aaa01a8139fd8bfb23298495d455a9099e65a7b209bcd246b81 WatchSource:0}: Error finding container 348f803f2c104aaa01a8139fd8bfb23298495d455a9099e65a7b209bcd246b81: Status 404 returned error can't find the container with id 348f803f2c104aaa01a8139fd8bfb23298495d455a9099e65a7b209bcd246b81
Oct 30 23:02:08.202479 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:08.202445    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" event={"ID":"380609b9-e345-4539-a18e-b63024b9621e","Type":"ContainerStarted","Data":"e50d2736510dc94ae324fe3f16752b0a702ec97685f69f54dbd349df13614e93"}
Oct 30 23:02:08.202883 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:08.202501    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" event={"ID":"380609b9-e345-4539-a18e-b63024b9621e","Type":"ContainerStarted","Data":"348f803f2c104aaa01a8139fd8bfb23298495d455a9099e65a7b209bcd246b81"}
Oct 30 23:02:12.219532 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:12.219500    4947 generic.go:356] "Generic (PLEG): container finished" podID="380609b9-e345-4539-a18e-b63024b9621e" containerID="e50d2736510dc94ae324fe3f16752b0a702ec97685f69f54dbd349df13614e93" exitCode=0
Oct 30 23:02:12.219954 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:12.219550    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" event={"ID":"380609b9-e345-4539-a18e-b63024b9621e","Type":"ContainerDied","Data":"e50d2736510dc94ae324fe3f16752b0a702ec97685f69f54dbd349df13614e93"}
Oct 30 23:02:13.224994 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:13.224955    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" event={"ID":"380609b9-e345-4539-a18e-b63024b9621e","Type":"ContainerStarted","Data":"410e686d4710a0fd0c5c3103c3e646192d32ce06ee7f96441ae4d1b0d18af6a1"}
Oct 30 23:02:13.225450 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:13.225024    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" event={"ID":"380609b9-e345-4539-a18e-b63024b9621e","Type":"ContainerStarted","Data":"742baabd67ea8bc80e3951bbe05d68e478d8571b233f63db3aa024837815e6ca"}
Oct 30 23:02:13.225450 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:13.225210    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:13.396733 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:13.396672    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" podStartSLOduration=3.378799272 podStartE2EDuration="7.396653052s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.01785378s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:13.395952504 +0000 UTC m=+3231.395300133" watchObservedRunningTime="2026-10-30 23:02:13.396653052 +0000 UTC m=+3231.396000615"
Oct 30 23:02:13.954713 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:13.954672    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 23:02:14.229400 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:14.229301    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:14.230970 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:14.230943    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 23:02:15.233024 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:15.232985    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 23:02:18.153189 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:18.153143    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 23:02:20.238061 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:20.238030    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:02:20.238534 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:20.238504    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 23:02:28.158518 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:28.158489    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:02:29.048556 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:02:29.048520    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:02:30.238962 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:30.238921    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 23:02:33.806207 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:33.806174    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"]
Oct 30 23:02:33.806658 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:33.806426    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kserve-container" containerID="cri-o://7b2a72cdf5be97a40ef8bb68a53360e9003a51b505147bd90b6b2d532caa0e8a" gracePeriod=30
Oct 30 23:02:33.806658 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:33.806465    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kube-rbac-proxy" containerID="cri-o://ae2a0d86cebaa7282201b74fbd89601e71ccfae4f6f82840d0f232e6aed4001f" gracePeriod=30
Oct 30 23:02:33.950195 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:33.950155    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.73:8643/healthz\": dial tcp 10.132.0.73:8643: connect: connection refused"
Oct 30 23:02:34.317434 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:34.317399    4947 generic.go:356] "Generic (PLEG): container finished" podID="22a16a9c-f385-419b-938e-0732450d4903" containerID="ae2a0d86cebaa7282201b74fbd89601e71ccfae4f6f82840d0f232e6aed4001f" exitCode=0
Oct 30 23:02:34.317600 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:34.317471    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" event={"ID":"22a16a9c-f385-419b-938e-0732450d4903","Type":"ContainerDied","Data":"ae2a0d86cebaa7282201b74fbd89601e71ccfae4f6f82840d0f232e6aed4001f"}
Oct 30 23:02:38.153449 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:38.153405    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 23:02:38.660360 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:38.660320    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:02:38.779085 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:38.779006    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a16a9c-f385-419b-938e-0732450d4903-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22a16a9c-f385-419b-938e-0732450d4903-cabundle-cert\") pod \"22a16a9c-f385-419b-938e-0732450d4903\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") "
Oct 30 23:02:38.779085 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:38.779053    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22a16a9c-f385-419b-938e-0732450d4903-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a16a9c-f385-419b-938e-0732450d4903-kserve-provision-location\") pod \"22a16a9c-f385-419b-938e-0732450d4903\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") "
Oct 30 23:02:38.779085 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:38.779079    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a16a9c-f385-419b-938e-0732450d4903-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22a16a9c-f385-419b-938e-0732450d4903-kube-rbac-proxy-sar-config\") pod \"22a16a9c-f385-419b-938e-0732450d4903\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") "
Oct 30 23:02:38.779510 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:38.779098    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a16a9c-f385-419b-938e-0732450d4903-kube-api-access-mlkhs\" (UniqueName: \"kubernetes.io/projected/22a16a9c-f385-419b-938e-0732450d4903-kube-api-access-mlkhs\") pod \"22a16a9c-f385-419b-938e-0732450d4903\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") "
Oct 30 23:02:38.779510 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:38.779136    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22a16a9c-f385-419b-938e-0732450d4903-proxy-tls\" (UniqueName: \"kubernetes.io/secret/22a16a9c-f385-419b-938e-0732450d4903-proxy-tls\") pod \"22a16a9c-f385-419b-938e-0732450d4903\" (UID: \"22a16a9c-f385-419b-938e-0732450d4903\") "
Oct 30 23:02:38.779510 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:38.779415    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a16a9c-f385-419b-938e-0732450d4903-kserve-provision-location" pod "22a16a9c-f385-419b-938e-0732450d4903" (UID: "22a16a9c-f385-419b-938e-0732450d4903"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:02:38.779510 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:38.779484    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a16a9c-f385-419b-938e-0732450d4903-cabundle-cert" pod "22a16a9c-f385-419b-938e-0732450d4903" (UID: "22a16a9c-f385-419b-938e-0732450d4903"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:38.779510 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:38.779493    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a16a9c-f385-419b-938e-0732450d4903-kube-rbac-proxy-sar-config" pod "22a16a9c-f385-419b-938e-0732450d4903" (UID: "22a16a9c-f385-419b-938e-0732450d4903"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:38.781387 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:38.781356    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a16a9c-f385-419b-938e-0732450d4903-kube-api-access-mlkhs" pod "22a16a9c-f385-419b-938e-0732450d4903" (UID: "22a16a9c-f385-419b-938e-0732450d4903"). InnerVolumeSpecName "kube-api-access-mlkhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:38.781495 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:38.781471    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a16a9c-f385-419b-938e-0732450d4903-proxy-tls" pod "22a16a9c-f385-419b-938e-0732450d4903" (UID: "22a16a9c-f385-419b-938e-0732450d4903"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:38.880246 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:38.880208    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22a16a9c-f385-419b-938e-0732450d4903-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:02:38.880246 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:38.880240    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a16a9c-f385-419b-938e-0732450d4903-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:02:38.880246 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:38.880252    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22a16a9c-f385-419b-938e-0732450d4903-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:02:38.880558 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:38.880261    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlkhs\" (UniqueName: \"kubernetes.io/projected/22a16a9c-f385-419b-938e-0732450d4903-kube-api-access-mlkhs\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:02:38.880558 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:38.880271    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22a16a9c-f385-419b-938e-0732450d4903-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:02:39.341813 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:39.341781    4947 generic.go:356] "Generic (PLEG): container finished" podID="22a16a9c-f385-419b-938e-0732450d4903" containerID="7b2a72cdf5be97a40ef8bb68a53360e9003a51b505147bd90b6b2d532caa0e8a" exitCode=0
Oct 30 23:02:39.342223 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:39.341836    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" event={"ID":"22a16a9c-f385-419b-938e-0732450d4903","Type":"ContainerDied","Data":"7b2a72cdf5be97a40ef8bb68a53360e9003a51b505147bd90b6b2d532caa0e8a"}
Oct 30 23:02:39.342223 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:39.341899    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7" event={"ID":"22a16a9c-f385-419b-938e-0732450d4903","Type":"ContainerDied","Data":"a524e37aadd30cc24e2b945fa5c12bb032047a66762af3b02e667dcad0393794"}
Oct 30 23:02:39.342223 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:39.341844    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"
Oct 30 23:02:39.342223 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:39.341925    4947 scope.go:122] "RemoveContainer" containerID="ae2a0d86cebaa7282201b74fbd89601e71ccfae4f6f82840d0f232e6aed4001f"
Oct 30 23:02:39.351388 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:39.351369    4947 scope.go:122] "RemoveContainer" containerID="7b2a72cdf5be97a40ef8bb68a53360e9003a51b505147bd90b6b2d532caa0e8a"
Oct 30 23:02:39.359258 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:39.359240    4947 scope.go:122] "RemoveContainer" containerID="ca08996891db8f9129bf5593255cc7bf5723c7e68dab30c40a6b90f751352d92"
Oct 30 23:02:39.366245 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:39.366216    4947 scope.go:122] "RemoveContainer" containerID="ae2a0d86cebaa7282201b74fbd89601e71ccfae4f6f82840d0f232e6aed4001f"
Oct 30 23:02:39.366515 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:02:39.366493    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2a0d86cebaa7282201b74fbd89601e71ccfae4f6f82840d0f232e6aed4001f\": container with ID starting with ae2a0d86cebaa7282201b74fbd89601e71ccfae4f6f82840d0f232e6aed4001f not found: ID does not exist" containerID="ae2a0d86cebaa7282201b74fbd89601e71ccfae4f6f82840d0f232e6aed4001f"
Oct 30 23:02:39.366592 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:39.366528    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2a0d86cebaa7282201b74fbd89601e71ccfae4f6f82840d0f232e6aed4001f"} err="failed to get container status \"ae2a0d86cebaa7282201b74fbd89601e71ccfae4f6f82840d0f232e6aed4001f\": rpc error: code = NotFound desc = could not find container \"ae2a0d86cebaa7282201b74fbd89601e71ccfae4f6f82840d0f232e6aed4001f\": container with ID starting with ae2a0d86cebaa7282201b74fbd89601e71ccfae4f6f82840d0f232e6aed4001f not found: ID does not exist"
Oct 30 23:02:39.366592 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:39.366553    4947 scope.go:122] "RemoveContainer" containerID="7b2a72cdf5be97a40ef8bb68a53360e9003a51b505147bd90b6b2d532caa0e8a"
Oct 30 23:02:39.366780 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:02:39.366763    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b2a72cdf5be97a40ef8bb68a53360e9003a51b505147bd90b6b2d532caa0e8a\": container with ID starting with 7b2a72cdf5be97a40ef8bb68a53360e9003a51b505147bd90b6b2d532caa0e8a not found: ID does not exist" containerID="7b2a72cdf5be97a40ef8bb68a53360e9003a51b505147bd90b6b2d532caa0e8a"
Oct 30 23:02:39.366853 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:39.366788    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b2a72cdf5be97a40ef8bb68a53360e9003a51b505147bd90b6b2d532caa0e8a"} err="failed to get container status \"7b2a72cdf5be97a40ef8bb68a53360e9003a51b505147bd90b6b2d532caa0e8a\": rpc error: code = NotFound desc = could not find container \"7b2a72cdf5be97a40ef8bb68a53360e9003a51b505147bd90b6b2d532caa0e8a\": container with ID starting with 7b2a72cdf5be97a40ef8bb68a53360e9003a51b505147bd90b6b2d532caa0e8a not found: ID does not exist"
Oct 30 23:02:39.366853 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:39.366808    4947 scope.go:122] "RemoveContainer" containerID="ca08996891db8f9129bf5593255cc7bf5723c7e68dab30c40a6b90f751352d92"
Oct 30 23:02:39.367038 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:02:39.367020    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca08996891db8f9129bf5593255cc7bf5723c7e68dab30c40a6b90f751352d92\": container with ID starting with ca08996891db8f9129bf5593255cc7bf5723c7e68dab30c40a6b90f751352d92 not found: ID does not exist" containerID="ca08996891db8f9129bf5593255cc7bf5723c7e68dab30c40a6b90f751352d92"
Oct 30 23:02:39.367081 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:39.367042    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca08996891db8f9129bf5593255cc7bf5723c7e68dab30c40a6b90f751352d92"} err="failed to get container status \"ca08996891db8f9129bf5593255cc7bf5723c7e68dab30c40a6b90f751352d92\": rpc error: code = NotFound desc = could not find container \"ca08996891db8f9129bf5593255cc7bf5723c7e68dab30c40a6b90f751352d92\": container with ID starting with ca08996891db8f9129bf5593255cc7bf5723c7e68dab30c40a6b90f751352d92 not found: ID does not exist"
Oct 30 23:02:39.509740 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:39.509702    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"]
Oct 30 23:02:39.515323 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:39.515298    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-sklearn-s3-tls-serving-pass-predictor-6d78588ff9-tvdv7"]
Oct 30 23:02:40.157991 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:40.157960    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a16a9c-f385-419b-938e-0732450d4903" path="/var/lib/kubelet/pods/22a16a9c-f385-419b-938e-0732450d4903/volumes"
Oct 30 23:02:40.238696 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:40.238660    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 23:02:50.238574 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:02:50.238538    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 23:02:59.049810 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:02:59.049765    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:03:00.239002 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:00.238959    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 23:03:01.272370 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.272312    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"]
Oct 30 23:03:01.272735 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.272671    4947 cpu_manager.go:414] "containerMap: removing container" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kube-rbac-proxy"
Oct 30 23:03:01.272735 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.272682    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kube-rbac-proxy"
Oct 30 23:03:01.272735 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.272708    4947 cpu_manager.go:414] "containerMap: removing container" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="storage-initializer"
Oct 30 23:03:01.272735 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.272714    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="storage-initializer"
Oct 30 23:03:01.272735 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.272720    4947 cpu_manager.go:414] "containerMap: removing container" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kserve-container"
Oct 30 23:03:01.272735 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.272726    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kserve-container"
Oct 30 23:03:01.272923 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.272781    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kserve-container"
Oct 30 23:03:01.272923 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.272792    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a16a9c-f385-419b-938e-0732450d4903" containerName="kube-rbac-proxy"
Oct 30 23:03:01.275903 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.275887    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:01.349124 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.349093    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1347dbed-6317-4f07-b322-6727394d545a-kserve-provision-location\") pod \"isvc-paddle-predictor-694b686766-jpbkz\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") " pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:01.349278 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.349137    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8444v\" (UniqueName: \"kubernetes.io/projected/1347dbed-6317-4f07-b322-6727394d545a-kube-api-access-8444v\") pod \"isvc-paddle-predictor-694b686766-jpbkz\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") " pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:01.349278 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.349212    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1347dbed-6317-4f07-b322-6727394d545a-proxy-tls\") pod \"isvc-paddle-predictor-694b686766-jpbkz\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") " pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:01.349375 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.349281    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1347dbed-6317-4f07-b322-6727394d545a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-694b686766-jpbkz\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") " pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:01.349375 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.349307    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1347dbed-6317-4f07-b322-6727394d545a-cabundle-cert\") pod \"isvc-paddle-predictor-694b686766-jpbkz\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") " pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:01.378709 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.378684    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"seaweedfs-s3-creds\""
Oct 30 23:03:01.378864 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.378717    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-paddle-predictor-serving-cert\""
Oct 30 23:03:01.378864 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.378829    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"openshift-service-ca.crt\""
Oct 30 23:03:01.378864 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.378850    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"kube-root-ca.crt\""
Oct 30 23:03:01.378864 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.378860    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Oct 30 23:03:01.379030 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.378858    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-bc5112d610\"/\"default-dockercfg-nnb5w\""
Oct 30 23:03:01.379030 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.378877    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-bc5112d610\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 23:03:01.380666 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.380644    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"]
Oct 30 23:03:01.449874 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.449848    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1347dbed-6317-4f07-b322-6727394d545a-proxy-tls\") pod \"isvc-paddle-predictor-694b686766-jpbkz\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") " pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:01.449874 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.449877    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1347dbed-6317-4f07-b322-6727394d545a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-694b686766-jpbkz\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") " pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:01.450071 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.449896    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1347dbed-6317-4f07-b322-6727394d545a-cabundle-cert\") pod \"isvc-paddle-predictor-694b686766-jpbkz\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") " pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:01.450071 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.449966    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1347dbed-6317-4f07-b322-6727394d545a-kserve-provision-location\") pod \"isvc-paddle-predictor-694b686766-jpbkz\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") " pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:01.450071 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.450007    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8444v\" (UniqueName: \"kubernetes.io/projected/1347dbed-6317-4f07-b322-6727394d545a-kube-api-access-8444v\") pod \"isvc-paddle-predictor-694b686766-jpbkz\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") " pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:01.450473 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.450450    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1347dbed-6317-4f07-b322-6727394d545a-kserve-provision-location\") pod \"isvc-paddle-predictor-694b686766-jpbkz\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") " pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:01.450653 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.450632    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1347dbed-6317-4f07-b322-6727394d545a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-694b686766-jpbkz\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") " pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:01.450700 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.450664    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1347dbed-6317-4f07-b322-6727394d545a-cabundle-cert\") pod \"isvc-paddle-predictor-694b686766-jpbkz\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") " pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:01.452469 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.452452    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1347dbed-6317-4f07-b322-6727394d545a-proxy-tls\") pod \"isvc-paddle-predictor-694b686766-jpbkz\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") " pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:01.512849 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.512820    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8444v\" (UniqueName: \"kubernetes.io/projected/1347dbed-6317-4f07-b322-6727394d545a-kube-api-access-8444v\") pod \"isvc-paddle-predictor-694b686766-jpbkz\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") " pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:01.593033 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:01.592997    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:02.019027 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:02.019003    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"]
Oct 30 23:03:02.021857 ip-10-0-133-218 kubenswrapper[4947]: W1030 23:03:02.021821    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1347dbed_6317_4f07_b322_6727394d545a.slice/crio-75e6632bf9751b94ad467b5727df490d3bac51eff7a05db4a92b307dcbeeab2b WatchSource:0}: Error finding container 75e6632bf9751b94ad467b5727df490d3bac51eff7a05db4a92b307dcbeeab2b: Status 404 returned error can't find the container with id 75e6632bf9751b94ad467b5727df490d3bac51eff7a05db4a92b307dcbeeab2b
Oct 30 23:03:02.442411 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:02.442372    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz" event={"ID":"1347dbed-6317-4f07-b322-6727394d545a","Type":"ContainerStarted","Data":"bf5000f47a946f0325b6e4e2bbf3309b1da38bccd1f124a4a80ff92e943ec410"}
Oct 30 23:03:02.442821 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:02.442462    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz" event={"ID":"1347dbed-6317-4f07-b322-6727394d545a","Type":"ContainerStarted","Data":"75e6632bf9751b94ad467b5727df490d3bac51eff7a05db4a92b307dcbeeab2b"}
Oct 30 23:03:08.468096 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:08.468065    4947 generic.go:356] "Generic (PLEG): container finished" podID="1347dbed-6317-4f07-b322-6727394d545a" containerID="bf5000f47a946f0325b6e4e2bbf3309b1da38bccd1f124a4a80ff92e943ec410" exitCode=0
Oct 30 23:03:08.468494 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:08.468141    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz" event={"ID":"1347dbed-6317-4f07-b322-6727394d545a","Type":"ContainerDied","Data":"bf5000f47a946f0325b6e4e2bbf3309b1da38bccd1f124a4a80ff92e943ec410"}
Oct 30 23:03:10.239439 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:10.239394    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 23:03:20.239043 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:20.239002    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 23:03:21.535828 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:21.535799    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz" event={"ID":"1347dbed-6317-4f07-b322-6727394d545a","Type":"ContainerStarted","Data":"56cf91dbf0907eae1ff102aab8f3a0b71dd03cf364ae6ff1a0b6f683e08acc49"}
Oct 30 23:03:21.536194 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:21.535859    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz" event={"ID":"1347dbed-6317-4f07-b322-6727394d545a","Type":"ContainerStarted","Data":"e175c08c8649983414216132da497eafa5c6bbe00a8fa880a22a0aafd124f43b"}
Oct 30 23:03:21.536194 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:21.536143    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:21.536297 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:21.536205    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:21.537322 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:21.537298    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz" podUID="1347dbed-6317-4f07-b322-6727394d545a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 23:03:21.706423 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:21.706375    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz" podStartSLOduration=1.7197303609999999 podStartE2EDuration="20.706346678s" totalImagesPullingTime="12.686028063s" totalInitContainerRuntime="6.300588254s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:03:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:03:21.705887571 +0000 UTC m=+3299.705235146" watchObservedRunningTime="2026-10-30 23:03:21.706346678 +0000 UTC m=+3299.705694232"
Oct 30 23:03:22.448204 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:22.448172    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 23:03:22.458828 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:22.453641    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 23:03:22.539698 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:22.539661    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz" podUID="1347dbed-6317-4f07-b322-6727394d545a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 23:03:27.544558 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:27.544530    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:03:27.545164 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:27.545138    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz" podUID="1347dbed-6317-4f07-b322-6727394d545a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 23:03:29.049503 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:03:29.049466    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:03:30.238651 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:30.238610    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 23:03:37.545781 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:37.545737    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz" podUID="1347dbed-6317-4f07-b322-6727394d545a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 23:03:40.239104 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:40.239062    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 23:03:47.153423 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:47.153392    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:03:47.545826 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:47.545791    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz" podUID="1347dbed-6317-4f07-b322-6727394d545a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 23:03:57.545683 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:03:57.545647    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz" podUID="1347dbed-6317-4f07-b322-6727394d545a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 23:03:59.048782 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:03:59.048743    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:04:00.088708 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:00.088672    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"]
Oct 30 23:04:00.089159 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:00.088933    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kserve-container" containerID="cri-o://742baabd67ea8bc80e3951bbe05d68e478d8571b233f63db3aa024837815e6ca" gracePeriod=30
Oct 30 23:04:00.089159 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:00.089005    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kube-rbac-proxy" containerID="cri-o://410e686d4710a0fd0c5c3103c3e646192d32ce06ee7f96441ae4d1b0d18af6a1" gracePeriod=30
Oct 30 23:04:00.233583 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:00.233530    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.74:8643/healthz\": dial tcp 10.132.0.74:8643: connect: connection refused"
Oct 30 23:04:00.695740 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:00.695708    4947 generic.go:356] "Generic (PLEG): container finished" podID="380609b9-e345-4539-a18e-b63024b9621e" containerID="410e686d4710a0fd0c5c3103c3e646192d32ce06ee7f96441ae4d1b0d18af6a1" exitCode=0
Oct 30 23:04:00.695915 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:00.695745    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" event={"ID":"380609b9-e345-4539-a18e-b63024b9621e","Type":"ContainerDied","Data":"410e686d4710a0fd0c5c3103c3e646192d32ce06ee7f96441ae4d1b0d18af6a1"}
Oct 30 23:04:05.234219 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.234177    4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.74:8643/healthz\": dial tcp 10.132.0.74:8643: connect: connection refused"
Oct 30 23:04:05.629610 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.629587    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:04:05.691922 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.691889    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/380609b9-e345-4539-a18e-b63024b9621e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380609b9-e345-4539-a18e-b63024b9621e-kserve-provision-location\") pod \"380609b9-e345-4539-a18e-b63024b9621e\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") "
Oct 30 23:04:05.692104 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.691940    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/380609b9-e345-4539-a18e-b63024b9621e-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/380609b9-e345-4539-a18e-b63024b9621e-cabundle-cert\") pod \"380609b9-e345-4539-a18e-b63024b9621e\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") "
Oct 30 23:04:05.692104 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.691973    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/380609b9-e345-4539-a18e-b63024b9621e-proxy-tls\" (UniqueName: \"kubernetes.io/secret/380609b9-e345-4539-a18e-b63024b9621e-proxy-tls\") pod \"380609b9-e345-4539-a18e-b63024b9621e\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") "
Oct 30 23:04:05.692104 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.692006    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/380609b9-e345-4539-a18e-b63024b9621e-kube-api-access-zwd48\" (UniqueName: \"kubernetes.io/projected/380609b9-e345-4539-a18e-b63024b9621e-kube-api-access-zwd48\") pod \"380609b9-e345-4539-a18e-b63024b9621e\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") "
Oct 30 23:04:05.692104 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.692025    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/380609b9-e345-4539-a18e-b63024b9621e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/380609b9-e345-4539-a18e-b63024b9621e-kube-rbac-proxy-sar-config\") pod \"380609b9-e345-4539-a18e-b63024b9621e\" (UID: \"380609b9-e345-4539-a18e-b63024b9621e\") "
Oct 30 23:04:05.692420 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.692289    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380609b9-e345-4539-a18e-b63024b9621e-kserve-provision-location" pod "380609b9-e345-4539-a18e-b63024b9621e" (UID: "380609b9-e345-4539-a18e-b63024b9621e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:05.692521 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.692395    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380609b9-e345-4539-a18e-b63024b9621e-cabundle-cert" pod "380609b9-e345-4539-a18e-b63024b9621e" (UID: "380609b9-e345-4539-a18e-b63024b9621e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:05.692521 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.692489    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380609b9-e345-4539-a18e-b63024b9621e-kube-rbac-proxy-sar-config" pod "380609b9-e345-4539-a18e-b63024b9621e" (UID: "380609b9-e345-4539-a18e-b63024b9621e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:05.694306 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.694286    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/380609b9-e345-4539-a18e-b63024b9621e-proxy-tls" pod "380609b9-e345-4539-a18e-b63024b9621e" (UID: "380609b9-e345-4539-a18e-b63024b9621e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:05.694405 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.694303    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380609b9-e345-4539-a18e-b63024b9621e-kube-api-access-zwd48" pod "380609b9-e345-4539-a18e-b63024b9621e" (UID: "380609b9-e345-4539-a18e-b63024b9621e"). InnerVolumeSpecName "kube-api-access-zwd48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:05.720655 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.720628    4947 generic.go:356] "Generic (PLEG): container finished" podID="380609b9-e345-4539-a18e-b63024b9621e" containerID="742baabd67ea8bc80e3951bbe05d68e478d8571b233f63db3aa024837815e6ca" exitCode=0
Oct 30 23:04:05.720763 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.720686    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"
Oct 30 23:04:05.720763 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.720705    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" event={"ID":"380609b9-e345-4539-a18e-b63024b9621e","Type":"ContainerDied","Data":"742baabd67ea8bc80e3951bbe05d68e478d8571b233f63db3aa024837815e6ca"}
Oct 30 23:04:05.720860 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.720763    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd" event={"ID":"380609b9-e345-4539-a18e-b63024b9621e","Type":"ContainerDied","Data":"348f803f2c104aaa01a8139fd8bfb23298495d455a9099e65a7b209bcd246b81"}
Oct 30 23:04:05.720860 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.720794    4947 scope.go:122] "RemoveContainer" containerID="410e686d4710a0fd0c5c3103c3e646192d32ce06ee7f96441ae4d1b0d18af6a1"
Oct 30 23:04:05.729937 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.729915    4947 scope.go:122] "RemoveContainer" containerID="742baabd67ea8bc80e3951bbe05d68e478d8571b233f63db3aa024837815e6ca"
Oct 30 23:04:05.737838 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.737821    4947 scope.go:122] "RemoveContainer" containerID="e50d2736510dc94ae324fe3f16752b0a702ec97685f69f54dbd349df13614e93"
Oct 30 23:04:05.744990 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.744975    4947 scope.go:122] "RemoveContainer" containerID="410e686d4710a0fd0c5c3103c3e646192d32ce06ee7f96441ae4d1b0d18af6a1"
Oct 30 23:04:05.745266 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:04:05.745248    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410e686d4710a0fd0c5c3103c3e646192d32ce06ee7f96441ae4d1b0d18af6a1\": container with ID starting with 410e686d4710a0fd0c5c3103c3e646192d32ce06ee7f96441ae4d1b0d18af6a1 not found: ID does not exist" containerID="410e686d4710a0fd0c5c3103c3e646192d32ce06ee7f96441ae4d1b0d18af6a1"
Oct 30 23:04:05.745307 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.745276    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410e686d4710a0fd0c5c3103c3e646192d32ce06ee7f96441ae4d1b0d18af6a1"} err="failed to get container status \"410e686d4710a0fd0c5c3103c3e646192d32ce06ee7f96441ae4d1b0d18af6a1\": rpc error: code = NotFound desc = could not find container \"410e686d4710a0fd0c5c3103c3e646192d32ce06ee7f96441ae4d1b0d18af6a1\": container with ID starting with 410e686d4710a0fd0c5c3103c3e646192d32ce06ee7f96441ae4d1b0d18af6a1 not found: ID does not exist"
Oct 30 23:04:05.745307 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.745293    4947 scope.go:122] "RemoveContainer" containerID="742baabd67ea8bc80e3951bbe05d68e478d8571b233f63db3aa024837815e6ca"
Oct 30 23:04:05.745604 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:04:05.745584    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"742baabd67ea8bc80e3951bbe05d68e478d8571b233f63db3aa024837815e6ca\": container with ID starting with 742baabd67ea8bc80e3951bbe05d68e478d8571b233f63db3aa024837815e6ca not found: ID does not exist" containerID="742baabd67ea8bc80e3951bbe05d68e478d8571b233f63db3aa024837815e6ca"
Oct 30 23:04:05.745655 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.745609    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742baabd67ea8bc80e3951bbe05d68e478d8571b233f63db3aa024837815e6ca"} err="failed to get container status \"742baabd67ea8bc80e3951bbe05d68e478d8571b233f63db3aa024837815e6ca\": rpc error: code = NotFound desc = could not find container \"742baabd67ea8bc80e3951bbe05d68e478d8571b233f63db3aa024837815e6ca\": container with ID starting with 742baabd67ea8bc80e3951bbe05d68e478d8571b233f63db3aa024837815e6ca not found: ID does not exist"
Oct 30 23:04:05.745655 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.745624    4947 scope.go:122] "RemoveContainer" containerID="e50d2736510dc94ae324fe3f16752b0a702ec97685f69f54dbd349df13614e93"
Oct 30 23:04:05.745859 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:04:05.745845    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50d2736510dc94ae324fe3f16752b0a702ec97685f69f54dbd349df13614e93\": container with ID starting with e50d2736510dc94ae324fe3f16752b0a702ec97685f69f54dbd349df13614e93 not found: ID does not exist" containerID="e50d2736510dc94ae324fe3f16752b0a702ec97685f69f54dbd349df13614e93"
Oct 30 23:04:05.745909 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.745862    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50d2736510dc94ae324fe3f16752b0a702ec97685f69f54dbd349df13614e93"} err="failed to get container status \"e50d2736510dc94ae324fe3f16752b0a702ec97685f69f54dbd349df13614e93\": rpc error: code = NotFound desc = could not find container \"e50d2736510dc94ae324fe3f16752b0a702ec97685f69f54dbd349df13614e93\": container with ID starting with e50d2736510dc94ae324fe3f16752b0a702ec97685f69f54dbd349df13614e93 not found: ID does not exist"
Oct 30 23:04:05.793292 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.793265    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380609b9-e345-4539-a18e-b63024b9621e-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:04:05.793292 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.793291    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/380609b9-e345-4539-a18e-b63024b9621e-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:04:05.793480 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.793302    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/380609b9-e345-4539-a18e-b63024b9621e-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:04:05.793480 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.793311    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwd48\" (UniqueName: \"kubernetes.io/projected/380609b9-e345-4539-a18e-b63024b9621e-kube-api-access-zwd48\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:04:05.793480 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.793320    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/380609b9-e345-4539-a18e-b63024b9621e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:04:05.918517 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.918479    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"]
Oct 30 23:04:05.923870 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:05.923840    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-runtime-predictor-6fff8ff9d8-lntrd"]
Oct 30 23:04:06.158187 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:06.158111    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="380609b9-e345-4539-a18e-b63024b9621e" path="/var/lib/kubelet/pods/380609b9-e345-4539-a18e-b63024b9621e/volumes"
Oct 30 23:04:07.546318 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:07.546290    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:04:09.149606 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.149568    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"]
Oct 30 23:04:09.150001 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.149888    4947 cpu_manager.go:414] "containerMap: removing container" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="storage-initializer"
Oct 30 23:04:09.150001 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.149901    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="storage-initializer"
Oct 30 23:04:09.150001 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.149914    4947 cpu_manager.go:414] "containerMap: removing container" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kube-rbac-proxy"
Oct 30 23:04:09.150001 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.149919    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kube-rbac-proxy"
Oct 30 23:04:09.150001 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.149946    4947 cpu_manager.go:414] "containerMap: removing container" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kserve-container"
Oct 30 23:04:09.150001 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.149951    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kserve-container"
Oct 30 23:04:09.150001 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.149999    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kube-rbac-proxy"
Oct 30 23:04:09.150249 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.150011    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="380609b9-e345-4539-a18e-b63024b9621e" containerName="kserve-container"
Oct 30 23:04:09.153066 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.153044    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:09.222483 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.222449    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f600f03-4345-46c4-81ee-aa0f7451c87c-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:09.222651 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.222503    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f600f03-4345-46c4-81ee-aa0f7451c87c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:09.222651 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.222627    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rqvh\" (UniqueName: \"kubernetes.io/projected/1f600f03-4345-46c4-81ee-aa0f7451c87c-kube-api-access-5rqvh\") pod \"isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:09.222741 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.222700    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f600f03-4345-46c4-81ee-aa0f7451c87c-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:09.222785 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.222773    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f600f03-4345-46c4-81ee-aa0f7451c87c-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:09.256185 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.256137    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"seaweedfs-s3-creds\""
Oct 30 23:04:09.256396 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.256286    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"kube-root-ca.crt\""
Oct 30 23:04:09.256396 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.256292    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"openshift-service-ca.crt\""
Oct 30 23:04:09.256396 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.256292    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 23:04:09.256396 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.256374    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"default-dockercfg-s9cqw\""
Oct 30 23:04:09.256396 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.256380    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 23:04:09.256603 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.256392    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Oct 30 23:04:09.257905 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.257888    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"]
Oct 30 23:04:09.323576 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.323550    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f600f03-4345-46c4-81ee-aa0f7451c87c-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:09.323736 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.323593    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f600f03-4345-46c4-81ee-aa0f7451c87c-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:09.323736 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.323628    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f600f03-4345-46c4-81ee-aa0f7451c87c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:09.323736 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.323659    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rqvh\" (UniqueName: \"kubernetes.io/projected/1f600f03-4345-46c4-81ee-aa0f7451c87c-kube-api-access-5rqvh\") pod \"isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:09.323736 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.323707    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f600f03-4345-46c4-81ee-aa0f7451c87c-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:09.324112 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.324088    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f600f03-4345-46c4-81ee-aa0f7451c87c-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:09.324255 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.324237    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f600f03-4345-46c4-81ee-aa0f7451c87c-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:09.324400 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.324384    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f600f03-4345-46c4-81ee-aa0f7451c87c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:09.326151 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.326134    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f600f03-4345-46c4-81ee-aa0f7451c87c-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:09.386461 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.386430    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rqvh\" (UniqueName: \"kubernetes.io/projected/1f600f03-4345-46c4-81ee-aa0f7451c87c-kube-api-access-5rqvh\") pod \"isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") " pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:09.466002 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.465917    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:09.842128 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:09.842105    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"]
Oct 30 23:04:09.845047 ip-10-0-133-218 kubenswrapper[4947]: W1030 23:04:09.845009    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f600f03_4345_46c4_81ee_aa0f7451c87c.slice/crio-c4a8330566981c4eb92c54c92d626757de584db4a6356461bb84b0915eead9f2 WatchSource:0}: Error finding container c4a8330566981c4eb92c54c92d626757de584db4a6356461bb84b0915eead9f2: Status 404 returned error can't find the container with id c4a8330566981c4eb92c54c92d626757de584db4a6356461bb84b0915eead9f2
Oct 30 23:04:10.745423 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:10.745387    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj" event={"ID":"1f600f03-4345-46c4-81ee-aa0f7451c87c","Type":"ContainerStarted","Data":"dbcba791fff8135c2467855ccd629a4e8dbaa3f450494e46cdb3be3f2f4e67e7"}
Oct 30 23:04:10.745842 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:10.745443    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj" event={"ID":"1f600f03-4345-46c4-81ee-aa0f7451c87c","Type":"ContainerStarted","Data":"c4a8330566981c4eb92c54c92d626757de584db4a6356461bb84b0915eead9f2"}
Oct 30 23:04:14.763850 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:14.763815    4947 generic.go:356] "Generic (PLEG): container finished" podID="1f600f03-4345-46c4-81ee-aa0f7451c87c" containerID="dbcba791fff8135c2467855ccd629a4e8dbaa3f450494e46cdb3be3f2f4e67e7" exitCode=0
Oct 30 23:04:14.764298 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:14.763885    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj" event={"ID":"1f600f03-4345-46c4-81ee-aa0f7451c87c","Type":"ContainerDied","Data":"dbcba791fff8135c2467855ccd629a4e8dbaa3f450494e46cdb3be3f2f4e67e7"}
Oct 30 23:04:15.771401 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:15.771366    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj" event={"ID":"1f600f03-4345-46c4-81ee-aa0f7451c87c","Type":"ContainerStarted","Data":"3da4479ede18c5e6fa9a3f3d46dfc822177bac3b992b60c116a350cb8d1650b1"}
Oct 30 23:04:15.771791 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:15.771437    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj" event={"ID":"1f600f03-4345-46c4-81ee-aa0f7451c87c","Type":"ContainerStarted","Data":"70110d3c5864cd699749038faf7e39b9a94c266c14e9793905b498159a8ef6c6"}
Oct 30 23:04:15.771791 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:15.771628    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:15.893224 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:15.893161    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj" podStartSLOduration=2.709338694 podStartE2EDuration="6.893147091s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.183808397s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:04:15.892848528 +0000 UTC m=+3353.892196103" watchObservedRunningTime="2026-10-30 23:04:15.893147091 +0000 UTC m=+3353.892494654"
Oct 30 23:04:16.775885 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:16.775854    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:22.785032 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:22.784999    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:22.928396 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:22.928363    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"]
Oct 30 23:04:22.928621 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:22.928582    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz" podUID="1347dbed-6317-4f07-b322-6727394d545a" containerName="kserve-container" containerID="cri-o://e175c08c8649983414216132da497eafa5c6bbe00a8fa880a22a0aafd124f43b" gracePeriod=30
Oct 30 23:04:22.928702 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:22.928679    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz" podUID="1347dbed-6317-4f07-b322-6727394d545a" containerName="kube-rbac-proxy" containerID="cri-o://56cf91dbf0907eae1ff102aab8f3a0b71dd03cf364ae6ff1a0b6f683e08acc49" gracePeriod=30
Oct 30 23:04:23.806120 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:23.806082    4947 generic.go:356] "Generic (PLEG): container finished" podID="1347dbed-6317-4f07-b322-6727394d545a" containerID="56cf91dbf0907eae1ff102aab8f3a0b71dd03cf364ae6ff1a0b6f683e08acc49" exitCode=0
Oct 30 23:04:23.806542 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:23.806151    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz" event={"ID":"1347dbed-6317-4f07-b322-6727394d545a","Type":"ContainerDied","Data":"56cf91dbf0907eae1ff102aab8f3a0b71dd03cf364ae6ff1a0b6f683e08acc49"}
Oct 30 23:04:25.775304 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.775282    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:04:25.816673 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.816637    4947 generic.go:356] "Generic (PLEG): container finished" podID="1347dbed-6317-4f07-b322-6727394d545a" containerID="e175c08c8649983414216132da497eafa5c6bbe00a8fa880a22a0aafd124f43b" exitCode=0
Oct 30 23:04:25.816806 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.816694    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"
Oct 30 23:04:25.816806 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.816714    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz" event={"ID":"1347dbed-6317-4f07-b322-6727394d545a","Type":"ContainerDied","Data":"e175c08c8649983414216132da497eafa5c6bbe00a8fa880a22a0aafd124f43b"}
Oct 30 23:04:25.816806 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.816784    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz" event={"ID":"1347dbed-6317-4f07-b322-6727394d545a","Type":"ContainerDied","Data":"75e6632bf9751b94ad467b5727df490d3bac51eff7a05db4a92b307dcbeeab2b"}
Oct 30 23:04:25.816917 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.816823    4947 scope.go:122] "RemoveContainer" containerID="56cf91dbf0907eae1ff102aab8f3a0b71dd03cf364ae6ff1a0b6f683e08acc49"
Oct 30 23:04:25.827156 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.827135    4947 scope.go:122] "RemoveContainer" containerID="e175c08c8649983414216132da497eafa5c6bbe00a8fa880a22a0aafd124f43b"
Oct 30 23:04:25.835140 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.835124    4947 scope.go:122] "RemoveContainer" containerID="bf5000f47a946f0325b6e4e2bbf3309b1da38bccd1f124a4a80ff92e943ec410"
Oct 30 23:04:25.842378 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.842356    4947 scope.go:122] "RemoveContainer" containerID="56cf91dbf0907eae1ff102aab8f3a0b71dd03cf364ae6ff1a0b6f683e08acc49"
Oct 30 23:04:25.842612 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:04:25.842591    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56cf91dbf0907eae1ff102aab8f3a0b71dd03cf364ae6ff1a0b6f683e08acc49\": container with ID starting with 56cf91dbf0907eae1ff102aab8f3a0b71dd03cf364ae6ff1a0b6f683e08acc49 not found: ID does not exist" containerID="56cf91dbf0907eae1ff102aab8f3a0b71dd03cf364ae6ff1a0b6f683e08acc49"
Oct 30 23:04:25.842659 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.842621    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56cf91dbf0907eae1ff102aab8f3a0b71dd03cf364ae6ff1a0b6f683e08acc49"} err="failed to get container status \"56cf91dbf0907eae1ff102aab8f3a0b71dd03cf364ae6ff1a0b6f683e08acc49\": rpc error: code = NotFound desc = could not find container \"56cf91dbf0907eae1ff102aab8f3a0b71dd03cf364ae6ff1a0b6f683e08acc49\": container with ID starting with 56cf91dbf0907eae1ff102aab8f3a0b71dd03cf364ae6ff1a0b6f683e08acc49 not found: ID does not exist"
Oct 30 23:04:25.842659 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.842638    4947 scope.go:122] "RemoveContainer" containerID="e175c08c8649983414216132da497eafa5c6bbe00a8fa880a22a0aafd124f43b"
Oct 30 23:04:25.842892 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:04:25.842874    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e175c08c8649983414216132da497eafa5c6bbe00a8fa880a22a0aafd124f43b\": container with ID starting with e175c08c8649983414216132da497eafa5c6bbe00a8fa880a22a0aafd124f43b not found: ID does not exist" containerID="e175c08c8649983414216132da497eafa5c6bbe00a8fa880a22a0aafd124f43b"
Oct 30 23:04:25.842937 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.842899    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e175c08c8649983414216132da497eafa5c6bbe00a8fa880a22a0aafd124f43b"} err="failed to get container status \"e175c08c8649983414216132da497eafa5c6bbe00a8fa880a22a0aafd124f43b\": rpc error: code = NotFound desc = could not find container \"e175c08c8649983414216132da497eafa5c6bbe00a8fa880a22a0aafd124f43b\": container with ID starting with e175c08c8649983414216132da497eafa5c6bbe00a8fa880a22a0aafd124f43b not found: ID does not exist"
Oct 30 23:04:25.842937 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.842916    4947 scope.go:122] "RemoveContainer" containerID="bf5000f47a946f0325b6e4e2bbf3309b1da38bccd1f124a4a80ff92e943ec410"
Oct 30 23:04:25.843139 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:04:25.843123    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf5000f47a946f0325b6e4e2bbf3309b1da38bccd1f124a4a80ff92e943ec410\": container with ID starting with bf5000f47a946f0325b6e4e2bbf3309b1da38bccd1f124a4a80ff92e943ec410 not found: ID does not exist" containerID="bf5000f47a946f0325b6e4e2bbf3309b1da38bccd1f124a4a80ff92e943ec410"
Oct 30 23:04:25.843178 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.843147    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf5000f47a946f0325b6e4e2bbf3309b1da38bccd1f124a4a80ff92e943ec410"} err="failed to get container status \"bf5000f47a946f0325b6e4e2bbf3309b1da38bccd1f124a4a80ff92e943ec410\": rpc error: code = NotFound desc = could not find container \"bf5000f47a946f0325b6e4e2bbf3309b1da38bccd1f124a4a80ff92e943ec410\": container with ID starting with bf5000f47a946f0325b6e4e2bbf3309b1da38bccd1f124a4a80ff92e943ec410 not found: ID does not exist"
Oct 30 23:04:25.867825 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.867776    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1347dbed-6317-4f07-b322-6727394d545a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1347dbed-6317-4f07-b322-6727394d545a-kube-rbac-proxy-sar-config\") pod \"1347dbed-6317-4f07-b322-6727394d545a\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") "
Oct 30 23:04:25.867901 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.867838    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1347dbed-6317-4f07-b322-6727394d545a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1347dbed-6317-4f07-b322-6727394d545a-cabundle-cert\") pod \"1347dbed-6317-4f07-b322-6727394d545a\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") "
Oct 30 23:04:25.867901 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.867860    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1347dbed-6317-4f07-b322-6727394d545a-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1347dbed-6317-4f07-b322-6727394d545a-proxy-tls\") pod \"1347dbed-6317-4f07-b322-6727394d545a\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") "
Oct 30 23:04:25.867974 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.867901    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1347dbed-6317-4f07-b322-6727394d545a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1347dbed-6317-4f07-b322-6727394d545a-kserve-provision-location\") pod \"1347dbed-6317-4f07-b322-6727394d545a\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") "
Oct 30 23:04:25.867974 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.867930    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1347dbed-6317-4f07-b322-6727394d545a-kube-api-access-8444v\" (UniqueName: \"kubernetes.io/projected/1347dbed-6317-4f07-b322-6727394d545a-kube-api-access-8444v\") pod \"1347dbed-6317-4f07-b322-6727394d545a\" (UID: \"1347dbed-6317-4f07-b322-6727394d545a\") "
Oct 30 23:04:25.868145 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.868123    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1347dbed-6317-4f07-b322-6727394d545a-kube-rbac-proxy-sar-config" pod "1347dbed-6317-4f07-b322-6727394d545a" (UID: "1347dbed-6317-4f07-b322-6727394d545a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:25.868251 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.868222    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1347dbed-6317-4f07-b322-6727394d545a-cabundle-cert" pod "1347dbed-6317-4f07-b322-6727394d545a" (UID: "1347dbed-6317-4f07-b322-6727394d545a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:25.870026 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.870005    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1347dbed-6317-4f07-b322-6727394d545a-kube-api-access-8444v" pod "1347dbed-6317-4f07-b322-6727394d545a" (UID: "1347dbed-6317-4f07-b322-6727394d545a"). InnerVolumeSpecName "kube-api-access-8444v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:25.870101 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.870065    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1347dbed-6317-4f07-b322-6727394d545a-proxy-tls" pod "1347dbed-6317-4f07-b322-6727394d545a" (UID: "1347dbed-6317-4f07-b322-6727394d545a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:25.881510 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.881485    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1347dbed-6317-4f07-b322-6727394d545a-kserve-provision-location" pod "1347dbed-6317-4f07-b322-6727394d545a" (UID: "1347dbed-6317-4f07-b322-6727394d545a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:25.968502 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.968472    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1347dbed-6317-4f07-b322-6727394d545a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:04:25.968502 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.968500    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1347dbed-6317-4f07-b322-6727394d545a-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:04:25.968722 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.968515    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1347dbed-6317-4f07-b322-6727394d545a-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:04:25.968722 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.968529    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1347dbed-6317-4f07-b322-6727394d545a-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:04:25.968722 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:25.968538    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8444v\" (UniqueName: \"kubernetes.io/projected/1347dbed-6317-4f07-b322-6727394d545a-kube-api-access-8444v\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:04:26.244532 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:26.244496    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"]
Oct 30 23:04:26.249417 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:26.249393    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-bc5112d610/isvc-paddle-predictor-694b686766-jpbkz"]
Oct 30 23:04:28.158195 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:28.158164    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1347dbed-6317-4f07-b322-6727394d545a" path="/var/lib/kubelet/pods/1347dbed-6317-4f07-b322-6727394d545a/volumes"
Oct 30 23:04:29.048267 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:04:29.048232    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:04:52.788750 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:52.788717    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:04:59.048880 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:04:59.048846    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:04:59.402283 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:59.402207    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"]
Oct 30 23:04:59.402474 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:59.402446    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj" podUID="1f600f03-4345-46c4-81ee-aa0f7451c87c" containerName="kserve-container" containerID="cri-o://70110d3c5864cd699749038faf7e39b9a94c266c14e9793905b498159a8ef6c6" gracePeriod=30
Oct 30 23:04:59.402537 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:59.402485    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj" podUID="1f600f03-4345-46c4-81ee-aa0f7451c87c" containerName="kube-rbac-proxy" containerID="cri-o://3da4479ede18c5e6fa9a3f3d46dfc822177bac3b992b60c116a350cb8d1650b1" gracePeriod=30
Oct 30 23:04:59.957143 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:59.957106    4947 generic.go:356] "Generic (PLEG): container finished" podID="1f600f03-4345-46c4-81ee-aa0f7451c87c" containerID="3da4479ede18c5e6fa9a3f3d46dfc822177bac3b992b60c116a350cb8d1650b1" exitCode=0
Oct 30 23:04:59.957359 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:04:59.957155    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj" event={"ID":"1f600f03-4345-46c4-81ee-aa0f7451c87c","Type":"ContainerDied","Data":"3da4479ede18c5e6fa9a3f3d46dfc822177bac3b992b60c116a350cb8d1650b1"}
Oct 30 23:05:00.446988 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.446962    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:05:00.549760 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.549733    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f600f03-4345-46c4-81ee-aa0f7451c87c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f600f03-4345-46c4-81ee-aa0f7451c87c-cabundle-cert\") pod \"1f600f03-4345-46c4-81ee-aa0f7451c87c\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") "
Oct 30 23:05:00.549910 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.549833    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f600f03-4345-46c4-81ee-aa0f7451c87c-kube-api-access-5rqvh\" (UniqueName: \"kubernetes.io/projected/1f600f03-4345-46c4-81ee-aa0f7451c87c-kube-api-access-5rqvh\") pod \"1f600f03-4345-46c4-81ee-aa0f7451c87c\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") "
Oct 30 23:05:00.549910 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.549858    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f600f03-4345-46c4-81ee-aa0f7451c87c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f600f03-4345-46c4-81ee-aa0f7451c87c-kube-rbac-proxy-sar-config\") pod \"1f600f03-4345-46c4-81ee-aa0f7451c87c\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") "
Oct 30 23:05:00.549910 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.549887    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f600f03-4345-46c4-81ee-aa0f7451c87c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f600f03-4345-46c4-81ee-aa0f7451c87c-kserve-provision-location\") pod \"1f600f03-4345-46c4-81ee-aa0f7451c87c\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") "
Oct 30 23:05:00.550028 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.549966    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f600f03-4345-46c4-81ee-aa0f7451c87c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f600f03-4345-46c4-81ee-aa0f7451c87c-proxy-tls\") pod \"1f600f03-4345-46c4-81ee-aa0f7451c87c\" (UID: \"1f600f03-4345-46c4-81ee-aa0f7451c87c\") "
Oct 30 23:05:00.550261 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.550138    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f600f03-4345-46c4-81ee-aa0f7451c87c-cabundle-cert" pod "1f600f03-4345-46c4-81ee-aa0f7451c87c" (UID: "1f600f03-4345-46c4-81ee-aa0f7451c87c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:00.550261 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.550245    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f600f03-4345-46c4-81ee-aa0f7451c87c-kube-rbac-proxy-sar-config" pod "1f600f03-4345-46c4-81ee-aa0f7451c87c" (UID: "1f600f03-4345-46c4-81ee-aa0f7451c87c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:00.550395 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.550270    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f600f03-4345-46c4-81ee-aa0f7451c87c-kserve-provision-location" pod "1f600f03-4345-46c4-81ee-aa0f7451c87c" (UID: "1f600f03-4345-46c4-81ee-aa0f7451c87c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:00.552161 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.552131    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f600f03-4345-46c4-81ee-aa0f7451c87c-proxy-tls" pod "1f600f03-4345-46c4-81ee-aa0f7451c87c" (UID: "1f600f03-4345-46c4-81ee-aa0f7451c87c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:00.552259 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.552180    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f600f03-4345-46c4-81ee-aa0f7451c87c-kube-api-access-5rqvh" pod "1f600f03-4345-46c4-81ee-aa0f7451c87c" (UID: "1f600f03-4345-46c4-81ee-aa0f7451c87c"). InnerVolumeSpecName "kube-api-access-5rqvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:00.651410 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.651377    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f600f03-4345-46c4-81ee-aa0f7451c87c-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:05:00.651410 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.651405    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f600f03-4345-46c4-81ee-aa0f7451c87c-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:05:00.651410 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.651415    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rqvh\" (UniqueName: \"kubernetes.io/projected/1f600f03-4345-46c4-81ee-aa0f7451c87c-kube-api-access-5rqvh\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:05:00.651645 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.651425    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f600f03-4345-46c4-81ee-aa0f7451c87c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:05:00.651645 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.651435    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f600f03-4345-46c4-81ee-aa0f7451c87c-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:05:00.962790 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.962700    4947 generic.go:356] "Generic (PLEG): container finished" podID="1f600f03-4345-46c4-81ee-aa0f7451c87c" containerID="70110d3c5864cd699749038faf7e39b9a94c266c14e9793905b498159a8ef6c6" exitCode=0
Oct 30 23:05:00.962790 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.962758    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"
Oct 30 23:05:00.962790 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.962774    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj" event={"ID":"1f600f03-4345-46c4-81ee-aa0f7451c87c","Type":"ContainerDied","Data":"70110d3c5864cd699749038faf7e39b9a94c266c14e9793905b498159a8ef6c6"}
Oct 30 23:05:00.963044 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.962831    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj" event={"ID":"1f600f03-4345-46c4-81ee-aa0f7451c87c","Type":"ContainerDied","Data":"c4a8330566981c4eb92c54c92d626757de584db4a6356461bb84b0915eead9f2"}
Oct 30 23:05:00.963044 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.962856    4947 scope.go:122] "RemoveContainer" containerID="3da4479ede18c5e6fa9a3f3d46dfc822177bac3b992b60c116a350cb8d1650b1"
Oct 30 23:05:00.974252 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.974231    4947 scope.go:122] "RemoveContainer" containerID="70110d3c5864cd699749038faf7e39b9a94c266c14e9793905b498159a8ef6c6"
Oct 30 23:05:00.982072 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.982053    4947 scope.go:122] "RemoveContainer" containerID="dbcba791fff8135c2467855ccd629a4e8dbaa3f450494e46cdb3be3f2f4e67e7"
Oct 30 23:05:00.989456 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.989431    4947 scope.go:122] "RemoveContainer" containerID="3da4479ede18c5e6fa9a3f3d46dfc822177bac3b992b60c116a350cb8d1650b1"
Oct 30 23:05:00.989723 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:05:00.989695    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da4479ede18c5e6fa9a3f3d46dfc822177bac3b992b60c116a350cb8d1650b1\": container with ID starting with 3da4479ede18c5e6fa9a3f3d46dfc822177bac3b992b60c116a350cb8d1650b1 not found: ID does not exist" containerID="3da4479ede18c5e6fa9a3f3d46dfc822177bac3b992b60c116a350cb8d1650b1"
Oct 30 23:05:00.989773 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.989733    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da4479ede18c5e6fa9a3f3d46dfc822177bac3b992b60c116a350cb8d1650b1"} err="failed to get container status \"3da4479ede18c5e6fa9a3f3d46dfc822177bac3b992b60c116a350cb8d1650b1\": rpc error: code = NotFound desc = could not find container \"3da4479ede18c5e6fa9a3f3d46dfc822177bac3b992b60c116a350cb8d1650b1\": container with ID starting with 3da4479ede18c5e6fa9a3f3d46dfc822177bac3b992b60c116a350cb8d1650b1 not found: ID does not exist"
Oct 30 23:05:00.989773 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.989751    4947 scope.go:122] "RemoveContainer" containerID="70110d3c5864cd699749038faf7e39b9a94c266c14e9793905b498159a8ef6c6"
Oct 30 23:05:00.989947 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:05:00.989932    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70110d3c5864cd699749038faf7e39b9a94c266c14e9793905b498159a8ef6c6\": container with ID starting with 70110d3c5864cd699749038faf7e39b9a94c266c14e9793905b498159a8ef6c6 not found: ID does not exist" containerID="70110d3c5864cd699749038faf7e39b9a94c266c14e9793905b498159a8ef6c6"
Oct 30 23:05:00.989986 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.989950    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70110d3c5864cd699749038faf7e39b9a94c266c14e9793905b498159a8ef6c6"} err="failed to get container status \"70110d3c5864cd699749038faf7e39b9a94c266c14e9793905b498159a8ef6c6\": rpc error: code = NotFound desc = could not find container \"70110d3c5864cd699749038faf7e39b9a94c266c14e9793905b498159a8ef6c6\": container with ID starting with 70110d3c5864cd699749038faf7e39b9a94c266c14e9793905b498159a8ef6c6 not found: ID does not exist"
Oct 30 23:05:00.989986 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.989971    4947 scope.go:122] "RemoveContainer" containerID="dbcba791fff8135c2467855ccd629a4e8dbaa3f450494e46cdb3be3f2f4e67e7"
Oct 30 23:05:00.990208 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:05:00.990193    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbcba791fff8135c2467855ccd629a4e8dbaa3f450494e46cdb3be3f2f4e67e7\": container with ID starting with dbcba791fff8135c2467855ccd629a4e8dbaa3f450494e46cdb3be3f2f4e67e7 not found: ID does not exist" containerID="dbcba791fff8135c2467855ccd629a4e8dbaa3f450494e46cdb3be3f2f4e67e7"
Oct 30 23:05:00.990261 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:00.990211    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbcba791fff8135c2467855ccd629a4e8dbaa3f450494e46cdb3be3f2f4e67e7"} err="failed to get container status \"dbcba791fff8135c2467855ccd629a4e8dbaa3f450494e46cdb3be3f2f4e67e7\": rpc error: code = NotFound desc = could not find container \"dbcba791fff8135c2467855ccd629a4e8dbaa3f450494e46cdb3be3f2f4e67e7\": container with ID starting with dbcba791fff8135c2467855ccd629a4e8dbaa3f450494e46cdb3be3f2f4e67e7 not found: ID does not exist"
Oct 30 23:05:01.139036 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:01.138999    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"]
Oct 30 23:05:01.188865 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:01.188831    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-lightgbm-v2-runtime-predictor-74ddff54b8-w6xbj"]
Oct 30 23:05:02.158239 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:02.158205    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f600f03-4345-46c4-81ee-aa0f7451c87c" path="/var/lib/kubelet/pods/1f600f03-4345-46c4-81ee-aa0f7451c87c/volumes"
Oct 30 23:05:29.048096 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:05:29.048052    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:05:49.609066 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.609032    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"]
Oct 30 23:05:49.609629 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.609509    4947 cpu_manager.go:414] "containerMap: removing container" podUID="1f600f03-4345-46c4-81ee-aa0f7451c87c" containerName="storage-initializer"
Oct 30 23:05:49.609629 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.609529    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f600f03-4345-46c4-81ee-aa0f7451c87c" containerName="storage-initializer"
Oct 30 23:05:49.609629 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.609550    4947 cpu_manager.go:414] "containerMap: removing container" podUID="1347dbed-6317-4f07-b322-6727394d545a" containerName="storage-initializer"
Oct 30 23:05:49.609629 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.609557    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1347dbed-6317-4f07-b322-6727394d545a" containerName="storage-initializer"
Oct 30 23:05:49.609629 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.609572    4947 cpu_manager.go:414] "containerMap: removing container" podUID="1f600f03-4345-46c4-81ee-aa0f7451c87c" containerName="kube-rbac-proxy"
Oct 30 23:05:49.609629 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.609580    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f600f03-4345-46c4-81ee-aa0f7451c87c" containerName="kube-rbac-proxy"
Oct 30 23:05:49.609629 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.609600    4947 cpu_manager.go:414] "containerMap: removing container" podUID="1347dbed-6317-4f07-b322-6727394d545a" containerName="kserve-container"
Oct 30 23:05:49.609629 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.609608    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1347dbed-6317-4f07-b322-6727394d545a" containerName="kserve-container"
Oct 30 23:05:49.610055 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.609649    4947 cpu_manager.go:414] "containerMap: removing container" podUID="1f600f03-4345-46c4-81ee-aa0f7451c87c" containerName="kserve-container"
Oct 30 23:05:49.610055 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.609657    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f600f03-4345-46c4-81ee-aa0f7451c87c" containerName="kserve-container"
Oct 30 23:05:49.610055 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.609670    4947 cpu_manager.go:414] "containerMap: removing container" podUID="1347dbed-6317-4f07-b322-6727394d545a" containerName="kube-rbac-proxy"
Oct 30 23:05:49.610055 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.609678    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1347dbed-6317-4f07-b322-6727394d545a" containerName="kube-rbac-proxy"
Oct 30 23:05:49.610055 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.609756    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f600f03-4345-46c4-81ee-aa0f7451c87c" containerName="kube-rbac-proxy"
Oct 30 23:05:49.610055 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.609769    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="1347dbed-6317-4f07-b322-6727394d545a" containerName="kserve-container"
Oct 30 23:05:49.610055 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.609781    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f600f03-4345-46c4-81ee-aa0f7451c87c" containerName="kserve-container"
Oct 30 23:05:49.610055 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.609793    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="1347dbed-6317-4f07-b322-6727394d545a" containerName="kube-rbac-proxy"
Oct 30 23:05:49.613116 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.613093    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:49.669365 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.669317    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 23:05:49.669729 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.669692    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"kube-root-ca.crt\""
Oct 30 23:05:49.669817 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.669762    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 23:05:49.695366 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.695321    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/688f0368-8944-4d74-a37e-6b2d2d1ec03f-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") " pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:49.695508 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.695420    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sbzm\" (UniqueName: \"kubernetes.io/projected/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kube-api-access-7sbzm\") pod \"isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") " pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:49.695508 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.695483    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") " pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:49.695624 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.695512    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/688f0368-8944-4d74-a37e-6b2d2d1ec03f-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") " pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:49.695682 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.695630    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") " pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:49.719473 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.719447    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Oct 30 23:05:49.719615 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.719521    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-959ee2f135\"/\"openshift-service-ca.crt\""
Oct 30 23:05:49.719615 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.719543    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"seaweedfs-s3-creds\""
Oct 30 23:05:49.719615 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.719547    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-959ee2f135\"/\"default-dockercfg-s9cqw\""
Oct 30 23:05:49.720764 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.720746    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"]
Oct 30 23:05:49.796116 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.796086    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/688f0368-8944-4d74-a37e-6b2d2d1ec03f-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") " pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:49.796273 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.796126    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") " pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:49.796273 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.796168    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/688f0368-8944-4d74-a37e-6b2d2d1ec03f-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") " pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:49.796273 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.796195    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sbzm\" (UniqueName: \"kubernetes.io/projected/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kube-api-access-7sbzm\") pod \"isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") " pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:49.796273 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.796251    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") " pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:49.796794 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.796769    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") " pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:49.797256 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.797227    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") " pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:49.797374 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.797355    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/688f0368-8944-4d74-a37e-6b2d2d1ec03f-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") " pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:49.798877 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.798859    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/688f0368-8944-4d74-a37e-6b2d2d1ec03f-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") " pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:49.862881 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.862823    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sbzm\" (UniqueName: \"kubernetes.io/projected/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kube-api-access-7sbzm\") pod \"isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") " pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:49.930974 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:49.930946    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:50.359168 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:50.359133    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"]
Oct 30 23:05:50.362269 ip-10-0-133-218 kubenswrapper[4947]: W1030 23:05:50.362240    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688f0368_8944_4d74_a37e_6b2d2d1ec03f.slice/crio-90d8c9ef9f2948db497bc10b81e31a9bae76689f7fcbb27b568c021232987571 WatchSource:0}: Error finding container 90d8c9ef9f2948db497bc10b81e31a9bae76689f7fcbb27b568c021232987571: Status 404 returned error can't find the container with id 90d8c9ef9f2948db497bc10b81e31a9bae76689f7fcbb27b568c021232987571
Oct 30 23:05:51.168194 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:51.168160    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4" event={"ID":"688f0368-8944-4d74-a37e-6b2d2d1ec03f","Type":"ContainerStarted","Data":"c392c72fcd80aecde7f4ac780252fb2aca20a75497c7ef5c75f06eec3ecb3e57"}
Oct 30 23:05:51.168603 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:51.168215    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4" event={"ID":"688f0368-8944-4d74-a37e-6b2d2d1ec03f","Type":"ContainerStarted","Data":"90d8c9ef9f2948db497bc10b81e31a9bae76689f7fcbb27b568c021232987571"}
Oct 30 23:05:55.189889 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:55.189854    4947 generic.go:356] "Generic (PLEG): container finished" podID="688f0368-8944-4d74-a37e-6b2d2d1ec03f" containerID="c392c72fcd80aecde7f4ac780252fb2aca20a75497c7ef5c75f06eec3ecb3e57" exitCode=0
Oct 30 23:05:55.190348 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:55.189930    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4" event={"ID":"688f0368-8944-4d74-a37e-6b2d2d1ec03f","Type":"ContainerDied","Data":"c392c72fcd80aecde7f4ac780252fb2aca20a75497c7ef5c75f06eec3ecb3e57"}
Oct 30 23:05:56.196524 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:56.196485    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4" event={"ID":"688f0368-8944-4d74-a37e-6b2d2d1ec03f","Type":"ContainerStarted","Data":"6e72e5250bbad06e2b4b039445dd74c2c56cab6e4b45c579f126fdf05ad3d6c6"}
Oct 30 23:05:56.196894 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:56.196549    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4" event={"ID":"688f0368-8944-4d74-a37e-6b2d2d1ec03f","Type":"ContainerStarted","Data":"91e34ae6aa5e2cb0da339c771a9937dfef2ce8615b4d96c83d2f7598d51fc091"}
Oct 30 23:05:56.196894 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:56.196864    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:56.196975 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:56.196910    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:05:56.321397 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:05:56.321305    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4" podStartSLOduration=2.727966033 podStartE2EDuration="7.321285987s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.593319954s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:56.320096439 +0000 UTC m=+3454.319444002" watchObservedRunningTime="2026-10-30 23:05:56.321285987 +0000 UTC m=+3454.320633550"
Oct 30 23:05:59.050547 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:05:59.050511    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:06:02.207004 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:02.206930    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:06:29.048173 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:06:29.048126    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:06:32.210723 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:32.210693    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:06:39.747169 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:39.747121    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"]
Oct 30 23:06:39.747609 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:39.747476    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4" podUID="688f0368-8944-4d74-a37e-6b2d2d1ec03f" containerName="kserve-container" containerID="cri-o://91e34ae6aa5e2cb0da339c771a9937dfef2ce8615b4d96c83d2f7598d51fc091" gracePeriod=30
Oct 30 23:06:39.747678 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:39.747582    4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4" podUID="688f0368-8944-4d74-a37e-6b2d2d1ec03f" containerName="kube-rbac-proxy" containerID="cri-o://6e72e5250bbad06e2b4b039445dd74c2c56cab6e4b45c579f126fdf05ad3d6c6" gracePeriod=30
Oct 30 23:06:40.375008 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:40.374975    4947 generic.go:356] "Generic (PLEG): container finished" podID="688f0368-8944-4d74-a37e-6b2d2d1ec03f" containerID="6e72e5250bbad06e2b4b039445dd74c2c56cab6e4b45c579f126fdf05ad3d6c6" exitCode=0
Oct 30 23:06:40.375210 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:40.375048    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4" event={"ID":"688f0368-8944-4d74-a37e-6b2d2d1ec03f","Type":"ContainerDied","Data":"6e72e5250bbad06e2b4b039445dd74c2c56cab6e4b45c579f126fdf05ad3d6c6"}
Oct 30 23:06:40.900643 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:40.900619    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:06:40.914789 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:40.914764    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/688f0368-8944-4d74-a37e-6b2d2d1ec03f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/688f0368-8944-4d74-a37e-6b2d2d1ec03f-cabundle-cert\") pod \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") "
Oct 30 23:06:40.914883 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:40.914833    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kube-rbac-proxy-sar-config\") pod \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") "
Oct 30 23:06:40.914931 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:40.914881    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/688f0368-8944-4d74-a37e-6b2d2d1ec03f-proxy-tls\" (UniqueName: \"kubernetes.io/secret/688f0368-8944-4d74-a37e-6b2d2d1ec03f-proxy-tls\") pod \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") "
Oct 30 23:06:40.914931 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:40.914909    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kserve-provision-location\") pod \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") "
Oct 30 23:06:40.915041 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:40.914942    4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kube-api-access-7sbzm\" (UniqueName: \"kubernetes.io/projected/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kube-api-access-7sbzm\") pod \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\" (UID: \"688f0368-8944-4d74-a37e-6b2d2d1ec03f\") "
Oct 30 23:06:40.915116 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:40.915092    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/688f0368-8944-4d74-a37e-6b2d2d1ec03f-cabundle-cert" pod "688f0368-8944-4d74-a37e-6b2d2d1ec03f" (UID: "688f0368-8944-4d74-a37e-6b2d2d1ec03f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:06:40.915234 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:40.915212    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kube-rbac-proxy-sar-config" pod "688f0368-8944-4d74-a37e-6b2d2d1ec03f" (UID: "688f0368-8944-4d74-a37e-6b2d2d1ec03f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:06:40.915287 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:40.915263    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kserve-provision-location" pod "688f0368-8944-4d74-a37e-6b2d2d1ec03f" (UID: "688f0368-8944-4d74-a37e-6b2d2d1ec03f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:06:40.915322 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:40.915290    4947 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/688f0368-8944-4d74-a37e-6b2d2d1ec03f-cabundle-cert\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:06:40.917067 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:40.917047    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/688f0368-8944-4d74-a37e-6b2d2d1ec03f-proxy-tls" pod "688f0368-8944-4d74-a37e-6b2d2d1ec03f" (UID: "688f0368-8944-4d74-a37e-6b2d2d1ec03f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:06:40.917166 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:40.917113    4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kube-api-access-7sbzm" pod "688f0368-8944-4d74-a37e-6b2d2d1ec03f" (UID: "688f0368-8944-4d74-a37e-6b2d2d1ec03f"). InnerVolumeSpecName "kube-api-access-7sbzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:06:41.016278 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.016203    4947 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/688f0368-8944-4d74-a37e-6b2d2d1ec03f-proxy-tls\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:06:41.016278 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.016234    4947 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kserve-provision-location\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:06:41.016278 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.016249    4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sbzm\" (UniqueName: \"kubernetes.io/projected/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kube-api-access-7sbzm\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:06:41.016278 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.016258    4947 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/688f0368-8944-4d74-a37e-6b2d2d1ec03f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-218.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:06:41.381596 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.381565    4947 generic.go:356] "Generic (PLEG): container finished" podID="688f0368-8944-4d74-a37e-6b2d2d1ec03f" containerID="91e34ae6aa5e2cb0da339c771a9937dfef2ce8615b4d96c83d2f7598d51fc091" exitCode=0
Oct 30 23:06:41.381797 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.381629    4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"
Oct 30 23:06:41.381797 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.381658    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4" event={"ID":"688f0368-8944-4d74-a37e-6b2d2d1ec03f","Type":"ContainerDied","Data":"91e34ae6aa5e2cb0da339c771a9937dfef2ce8615b4d96c83d2f7598d51fc091"}
Oct 30 23:06:41.381797 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.381730    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4" event={"ID":"688f0368-8944-4d74-a37e-6b2d2d1ec03f","Type":"ContainerDied","Data":"90d8c9ef9f2948db497bc10b81e31a9bae76689f7fcbb27b568c021232987571"}
Oct 30 23:06:41.381797 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.381766    4947 scope.go:122] "RemoveContainer" containerID="6e72e5250bbad06e2b4b039445dd74c2c56cab6e4b45c579f126fdf05ad3d6c6"
Oct 30 23:06:41.390507 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.390487    4947 scope.go:122] "RemoveContainer" containerID="91e34ae6aa5e2cb0da339c771a9937dfef2ce8615b4d96c83d2f7598d51fc091"
Oct 30 23:06:41.398644 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.398622    4947 scope.go:122] "RemoveContainer" containerID="c392c72fcd80aecde7f4ac780252fb2aca20a75497c7ef5c75f06eec3ecb3e57"
Oct 30 23:06:41.406215 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.406193    4947 scope.go:122] "RemoveContainer" containerID="6e72e5250bbad06e2b4b039445dd74c2c56cab6e4b45c579f126fdf05ad3d6c6"
Oct 30 23:06:41.406500 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:06:41.406479    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e72e5250bbad06e2b4b039445dd74c2c56cab6e4b45c579f126fdf05ad3d6c6\": container with ID starting with 6e72e5250bbad06e2b4b039445dd74c2c56cab6e4b45c579f126fdf05ad3d6c6 not found: ID does not exist" containerID="6e72e5250bbad06e2b4b039445dd74c2c56cab6e4b45c579f126fdf05ad3d6c6"
Oct 30 23:06:41.406552 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.406515    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e72e5250bbad06e2b4b039445dd74c2c56cab6e4b45c579f126fdf05ad3d6c6"} err="failed to get container status \"6e72e5250bbad06e2b4b039445dd74c2c56cab6e4b45c579f126fdf05ad3d6c6\": rpc error: code = NotFound desc = could not find container \"6e72e5250bbad06e2b4b039445dd74c2c56cab6e4b45c579f126fdf05ad3d6c6\": container with ID starting with 6e72e5250bbad06e2b4b039445dd74c2c56cab6e4b45c579f126fdf05ad3d6c6 not found: ID does not exist"
Oct 30 23:06:41.406552 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.406534    4947 scope.go:122] "RemoveContainer" containerID="91e34ae6aa5e2cb0da339c771a9937dfef2ce8615b4d96c83d2f7598d51fc091"
Oct 30 23:06:41.406798 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:06:41.406777    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e34ae6aa5e2cb0da339c771a9937dfef2ce8615b4d96c83d2f7598d51fc091\": container with ID starting with 91e34ae6aa5e2cb0da339c771a9937dfef2ce8615b4d96c83d2f7598d51fc091 not found: ID does not exist" containerID="91e34ae6aa5e2cb0da339c771a9937dfef2ce8615b4d96c83d2f7598d51fc091"
Oct 30 23:06:41.406852 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.406804    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e34ae6aa5e2cb0da339c771a9937dfef2ce8615b4d96c83d2f7598d51fc091"} err="failed to get container status \"91e34ae6aa5e2cb0da339c771a9937dfef2ce8615b4d96c83d2f7598d51fc091\": rpc error: code = NotFound desc = could not find container \"91e34ae6aa5e2cb0da339c771a9937dfef2ce8615b4d96c83d2f7598d51fc091\": container with ID starting with 91e34ae6aa5e2cb0da339c771a9937dfef2ce8615b4d96c83d2f7598d51fc091 not found: ID does not exist"
Oct 30 23:06:41.406852 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.406819    4947 scope.go:122] "RemoveContainer" containerID="c392c72fcd80aecde7f4ac780252fb2aca20a75497c7ef5c75f06eec3ecb3e57"
Oct 30 23:06:41.407021 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:06:41.407007    4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c392c72fcd80aecde7f4ac780252fb2aca20a75497c7ef5c75f06eec3ecb3e57\": container with ID starting with c392c72fcd80aecde7f4ac780252fb2aca20a75497c7ef5c75f06eec3ecb3e57 not found: ID does not exist" containerID="c392c72fcd80aecde7f4ac780252fb2aca20a75497c7ef5c75f06eec3ecb3e57"
Oct 30 23:06:41.407060 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.407035    4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c392c72fcd80aecde7f4ac780252fb2aca20a75497c7ef5c75f06eec3ecb3e57"} err="failed to get container status \"c392c72fcd80aecde7f4ac780252fb2aca20a75497c7ef5c75f06eec3ecb3e57\": rpc error: code = NotFound desc = could not find container \"c392c72fcd80aecde7f4ac780252fb2aca20a75497c7ef5c75f06eec3ecb3e57\": container with ID starting with c392c72fcd80aecde7f4ac780252fb2aca20a75497c7ef5c75f06eec3ecb3e57 not found: ID does not exist"
Oct 30 23:06:41.550200 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.550167    4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"]
Oct 30 23:06:41.555539 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:41.555509    4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-959ee2f135/isvc-mlflow-v2-runtime-predictor-f657c4849-kkxb4"]
Oct 30 23:06:42.158626 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:06:42.158593    4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="688f0368-8944-4d74-a37e-6b2d2d1ec03f" path="/var/lib/kubelet/pods/688f0368-8944-4d74-a37e-6b2d2d1ec03f/volumes"
Oct 30 23:06:59.048268 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:06:59.048231    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:07:29.048436 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:07:29.048400    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:07:59.048586 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:07:59.048547    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:08:22.478913 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:08:22.478883    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 23:08:22.484506 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:08:22.484485    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 23:08:29.048833 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:08:29.048799    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:08:59.048189 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:08:59.048145    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:09:29.050250 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:09:29.050198    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:09:59.048484 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:09:59.048452    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:10:29.047979 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:10:29.047944    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:10:59.048796 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:10:59.048713    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:11:29.048605 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:11:29.048498    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:11:43.077018 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.076984    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dk65g/must-gather-p89tj"]
Oct 30 23:11:43.077451 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.077294    4947 cpu_manager.go:414] "containerMap: removing container" podUID="688f0368-8944-4d74-a37e-6b2d2d1ec03f" containerName="storage-initializer"
Oct 30 23:11:43.077451 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.077307    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="688f0368-8944-4d74-a37e-6b2d2d1ec03f" containerName="storage-initializer"
Oct 30 23:11:43.077451 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.077319    4947 cpu_manager.go:414] "containerMap: removing container" podUID="688f0368-8944-4d74-a37e-6b2d2d1ec03f" containerName="kserve-container"
Oct 30 23:11:43.077451 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.077324    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="688f0368-8944-4d74-a37e-6b2d2d1ec03f" containerName="kserve-container"
Oct 30 23:11:43.077451 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.077355    4947 cpu_manager.go:414] "containerMap: removing container" podUID="688f0368-8944-4d74-a37e-6b2d2d1ec03f" containerName="kube-rbac-proxy"
Oct 30 23:11:43.077451 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.077362    4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="688f0368-8944-4d74-a37e-6b2d2d1ec03f" containerName="kube-rbac-proxy"
Oct 30 23:11:43.077451 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.077426    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="688f0368-8944-4d74-a37e-6b2d2d1ec03f" containerName="kserve-container"
Oct 30 23:11:43.077451 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.077435    4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="688f0368-8944-4d74-a37e-6b2d2d1ec03f" containerName="kube-rbac-proxy"
Oct 30 23:11:43.080471 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.080443    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dk65g/must-gather-p89tj"
Oct 30 23:11:43.134240 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.134207    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dk65g\"/\"kube-root-ca.crt\""
Oct 30 23:11:43.183005 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.182974    4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dk65g\"/\"default-dockercfg-w2m2f\""
Oct 30 23:11:43.183005 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.183003    4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dk65g\"/\"openshift-service-ca.crt\""
Oct 30 23:11:43.184827 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.184806    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dk65g/must-gather-p89tj"]
Oct 30 23:11:43.240065 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.240035    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb5gr\" (UniqueName: \"kubernetes.io/projected/d8656467-689b-41ea-ae91-b333ad14a2cc-kube-api-access-fb5gr\") pod \"must-gather-p89tj\" (UID: \"d8656467-689b-41ea-ae91-b333ad14a2cc\") " pod="openshift-must-gather-dk65g/must-gather-p89tj"
Oct 30 23:11:43.240235 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.240094    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d8656467-689b-41ea-ae91-b333ad14a2cc-must-gather-output\") pod \"must-gather-p89tj\" (UID: \"d8656467-689b-41ea-ae91-b333ad14a2cc\") " pod="openshift-must-gather-dk65g/must-gather-p89tj"
Oct 30 23:11:43.340744 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.340661    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d8656467-689b-41ea-ae91-b333ad14a2cc-must-gather-output\") pod \"must-gather-p89tj\" (UID: \"d8656467-689b-41ea-ae91-b333ad14a2cc\") " pod="openshift-must-gather-dk65g/must-gather-p89tj"
Oct 30 23:11:43.340744 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.340740    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb5gr\" (UniqueName: \"kubernetes.io/projected/d8656467-689b-41ea-ae91-b333ad14a2cc-kube-api-access-fb5gr\") pod \"must-gather-p89tj\" (UID: \"d8656467-689b-41ea-ae91-b333ad14a2cc\") " pod="openshift-must-gather-dk65g/must-gather-p89tj"
Oct 30 23:11:43.341020 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.341000    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d8656467-689b-41ea-ae91-b333ad14a2cc-must-gather-output\") pod \"must-gather-p89tj\" (UID: \"d8656467-689b-41ea-ae91-b333ad14a2cc\") " pod="openshift-must-gather-dk65g/must-gather-p89tj"
Oct 30 23:11:43.403557 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.403523    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb5gr\" (UniqueName: \"kubernetes.io/projected/d8656467-689b-41ea-ae91-b333ad14a2cc-kube-api-access-fb5gr\") pod \"must-gather-p89tj\" (UID: \"d8656467-689b-41ea-ae91-b333ad14a2cc\") " pod="openshift-must-gather-dk65g/must-gather-p89tj"
Oct 30 23:11:43.406304 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.406283    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dk65g/must-gather-p89tj"
Oct 30 23:11:43.779073 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.778999    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dk65g/must-gather-p89tj"]
Oct 30 23:11:43.782464 ip-10-0-133-218 kubenswrapper[4947]: W1030 23:11:43.782436    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8656467_689b_41ea_ae91_b333ad14a2cc.slice/crio-f916a32cd08e34b0a4de26193a6e569e14131296bde587265e70a785d58ed7a0 WatchSource:0}: Error finding container f916a32cd08e34b0a4de26193a6e569e14131296bde587265e70a785d58ed7a0: Status 404 returned error can't find the container with id f916a32cd08e34b0a4de26193a6e569e14131296bde587265e70a785d58ed7a0
Oct 30 23:11:43.784190 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:43.784172    4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:11:44.567320 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:44.567287    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dk65g/must-gather-p89tj" event={"ID":"d8656467-689b-41ea-ae91-b333ad14a2cc","Type":"ContainerStarted","Data":"f916a32cd08e34b0a4de26193a6e569e14131296bde587265e70a785d58ed7a0"}
Oct 30 23:11:45.573251 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:45.573213    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dk65g/must-gather-p89tj" event={"ID":"d8656467-689b-41ea-ae91-b333ad14a2cc","Type":"ContainerStarted","Data":"97cff7b1311db83d6e4caee2385d56b545c399bd8eb2cc6e9cdaf388094d574f"}
Oct 30 23:11:45.573680 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:45.573261    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dk65g/must-gather-p89tj" event={"ID":"d8656467-689b-41ea-ae91-b333ad14a2cc","Type":"ContainerStarted","Data":"42ef0b3acf072bab1bdd0ea036bcb9e37627b9d0189e25cc9380918cac8bfff0"}
Oct 30 23:11:45.690772 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:45.690719    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dk65g/must-gather-p89tj" podStartSLOduration=1.432715199 podStartE2EDuration="2.690704368s" totalImagesPullingTime="1.257989169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:11:45.690099868 +0000 UTC m=+3803.689447422" watchObservedRunningTime="2026-10-30 23:11:45.690704368 +0000 UTC m=+3803.690051931"
Oct 30 23:11:52.882467 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:52.882434    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xtgwp_93f981a6-4af6-4405-aec6-89e539c42c67/global-pull-secret-syncer/0.log"
Oct 30 23:11:54.116363 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:54.112352    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6dsmb_b9d3d620-4040-45ca-86da-ce6ba161ba3a/connection-checker/0.log"
Oct 30 23:11:54.543157 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:54.543129    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-79956_ba0fc359-3a19-49ea-b798-2ca0080b30b9/connection-checker/0.log"
Oct 30 23:11:54.992211 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:54.992141    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jktvv_fcb4abba-e59f-4e64-af78-f1e60697b4b2/connection-checker/0.log"
Oct 30 23:11:56.639557 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:56.639523    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fgkn6_2188065c-aa72-45ea-a0bb-9665fe8f51a2/konnectivity-agent/0.log"
Oct 30 23:11:57.455512 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:11:57.455480    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-218.us-west-1.compute.internal_aaa975e9f9cee63edf4d69aef6d66c72/haproxy/0.log"
Oct 30 23:11:59.049247 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:11:59.049214    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"
Oct 30 23:12:04.952189 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:04.952108    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-k5qrx_557fc88c-2102-45ec-adc4-80bae4f140cb/cluster-monitoring-operator/0.log"
Oct 30 23:12:05.955582 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:05.955550    4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"]
Oct 30 23:12:05.960870 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:05.960844    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:06.012889 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.012855    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f83dfc1-cade-4e3d-9809-c446229b0f69-proc\") pod \"perf-node-gather-daemonset-zjmw9\" (UID: \"0f83dfc1-cade-4e3d-9809-c446229b0f69\") " pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:06.013057 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.012909    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f83dfc1-cade-4e3d-9809-c446229b0f69-podres\") pod \"perf-node-gather-daemonset-zjmw9\" (UID: \"0f83dfc1-cade-4e3d-9809-c446229b0f69\") " pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:06.013099 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.013055    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcvv5\" (UniqueName: \"kubernetes.io/projected/0f83dfc1-cade-4e3d-9809-c446229b0f69-kube-api-access-vcvv5\") pod \"perf-node-gather-daemonset-zjmw9\" (UID: \"0f83dfc1-cade-4e3d-9809-c446229b0f69\") " pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:06.013151 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.013103    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f83dfc1-cade-4e3d-9809-c446229b0f69-sys\") pod \"perf-node-gather-daemonset-zjmw9\" (UID: \"0f83dfc1-cade-4e3d-9809-c446229b0f69\") " pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:06.013198 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.013177    4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f83dfc1-cade-4e3d-9809-c446229b0f69-lib-modules\") pod \"perf-node-gather-daemonset-zjmw9\" (UID: \"0f83dfc1-cade-4e3d-9809-c446229b0f69\") " pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:06.019256 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.019225    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"]
Oct 30 23:12:06.114210 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.114176    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f83dfc1-cade-4e3d-9809-c446229b0f69-sys\") pod \"perf-node-gather-daemonset-zjmw9\" (UID: \"0f83dfc1-cade-4e3d-9809-c446229b0f69\") " pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:06.114210 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.114211    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f83dfc1-cade-4e3d-9809-c446229b0f69-lib-modules\") pod \"perf-node-gather-daemonset-zjmw9\" (UID: \"0f83dfc1-cade-4e3d-9809-c446229b0f69\") " pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:06.114439 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.114241    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f83dfc1-cade-4e3d-9809-c446229b0f69-proc\") pod \"perf-node-gather-daemonset-zjmw9\" (UID: \"0f83dfc1-cade-4e3d-9809-c446229b0f69\") " pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:06.114439 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.114301    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f83dfc1-cade-4e3d-9809-c446229b0f69-proc\") pod \"perf-node-gather-daemonset-zjmw9\" (UID: \"0f83dfc1-cade-4e3d-9809-c446229b0f69\") " pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:06.114439 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.114301    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f83dfc1-cade-4e3d-9809-c446229b0f69-sys\") pod \"perf-node-gather-daemonset-zjmw9\" (UID: \"0f83dfc1-cade-4e3d-9809-c446229b0f69\") " pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:06.114439 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.114323    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f83dfc1-cade-4e3d-9809-c446229b0f69-lib-modules\") pod \"perf-node-gather-daemonset-zjmw9\" (UID: \"0f83dfc1-cade-4e3d-9809-c446229b0f69\") " pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:06.114439 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.114375    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f83dfc1-cade-4e3d-9809-c446229b0f69-podres\") pod \"perf-node-gather-daemonset-zjmw9\" (UID: \"0f83dfc1-cade-4e3d-9809-c446229b0f69\") " pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:06.114439 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.114412    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f83dfc1-cade-4e3d-9809-c446229b0f69-podres\") pod \"perf-node-gather-daemonset-zjmw9\" (UID: \"0f83dfc1-cade-4e3d-9809-c446229b0f69\") " pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:06.114439 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.114428    4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcvv5\" (UniqueName: \"kubernetes.io/projected/0f83dfc1-cade-4e3d-9809-c446229b0f69-kube-api-access-vcvv5\") pod \"perf-node-gather-daemonset-zjmw9\" (UID: \"0f83dfc1-cade-4e3d-9809-c446229b0f69\") " pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:06.177096 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.177073    4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcvv5\" (UniqueName: \"kubernetes.io/projected/0f83dfc1-cade-4e3d-9809-c446229b0f69-kube-api-access-vcvv5\") pod \"perf-node-gather-daemonset-zjmw9\" (UID: \"0f83dfc1-cade-4e3d-9809-c446229b0f69\") " pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:06.276190 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.276123    4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:06.647806 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.647772    4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"]
Oct 30 23:12:06.651151 ip-10-0-133-218 kubenswrapper[4947]: W1030 23:12:06.651118    4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0f83dfc1_cade_4e3d_9809_c446229b0f69.slice/crio-5a2e0b1444106c03ef5cf3edb653455c74061e8021f02cc2dd8128f0e864afc2 WatchSource:0}: Error finding container 5a2e0b1444106c03ef5cf3edb653455c74061e8021f02cc2dd8128f0e864afc2: Status 404 returned error can't find the container with id 5a2e0b1444106c03ef5cf3edb653455c74061e8021f02cc2dd8128f0e864afc2
Oct 30 23:12:06.679249 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:06.679219    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9" event={"ID":"0f83dfc1-cade-4e3d-9809-c446229b0f69","Type":"ContainerStarted","Data":"5a2e0b1444106c03ef5cf3edb653455c74061e8021f02cc2dd8128f0e864afc2"}
Oct 30 23:12:07.685541 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:07.685499    4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9" event={"ID":"0f83dfc1-cade-4e3d-9809-c446229b0f69","Type":"ContainerStarted","Data":"1aebfda6e0a1e8fbf585c986b3586769c707761694a94fe15ae4d011d00e6b04"}
Oct 30 23:12:07.686209 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:07.685562    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:07.802278 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:07.802227    4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9" podStartSLOduration=2.802213955 podStartE2EDuration="2.802213955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:07.802073408 +0000 UTC m=+3825.801420970" watchObservedRunningTime="2026-10-30 23:12:07.802213955 +0000 UTC m=+3825.801561518"
Oct 30 23:12:09.990319 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:09.990276    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zcmwq_040c9e13-1604-4a8b-9507-cd2617a1e3e2/node-exporter/0.log"
Oct 30 23:12:10.405481 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:10.405443    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zcmwq_040c9e13-1604-4a8b-9507-cd2617a1e3e2/kube-rbac-proxy/0.log"
Oct 30 23:12:10.821902 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:10.821873    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zcmwq_040c9e13-1604-4a8b-9507-cd2617a1e3e2/init-textfile/0.log"
Oct 30 23:12:13.699869 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:13.699844    4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dk65g/perf-node-gather-daemonset-zjmw9"
Oct 30 23:12:22.932731 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:22.932704    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-8sk5t_ba52011d-6892-4b4d-ac14-07e8be859d9e/networking-console-plugin/0.log"
Oct 30 23:12:25.641440 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:25.641398    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/3.log"
Oct 30 23:12:25.648958 ip-10-0-133-218 kubenswrapper[4947]: I1030 23:12:25.648934    4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gs4xr_00c23fb6-cbfb-4498-ad17-15998e6d2d66/console-operator/4.log"
Oct 30 23:12:29.051305 ip-10-0-133-218 kubenswrapper[4947]: E1030 23:12:29.051259    4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dqpbw" containerName="extractor"